[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from HmsConfigurations/MasterStressTest
[ RUN      ] HmsConfigurations/MasterStressTest.Test/0
I0619 00:11:43.358011 25052 test_util.cc:208] Using random seed: 1693892628
I0619 00:11:43.388041 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.62:37587
--webserver_interface=127.24.119.62
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:11:43.542177 25059 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:11:43.542562 25059 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:11:43.551653 25059 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:11:43.551776 25059 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0619 00:11:43.551899 25059 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:11:43.552007 25059 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:11:43.552129 25059 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:11:43.571430 25059 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.62:37587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.119.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:11:43.580787 25059 master_main.cc:80] Initializing master server...
I0619 00:11:43.583027 25059 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 439913us
I0619 00:11:43.583766 25059 fs_manager.cc:261] Metadata directory not provided
I0619 00:11:43.583899 25059 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:11:43.584156 25059 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:11:43.584275 25059 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:11:43.593427 25059 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data/instance:
uuid: "cb25d1bba5bf4c64914f300f122ec898"
format_stamp: "Formatted at 2018-06-19 07:11:43 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:43.594245 25059 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/wal/instance:
uuid: "cb25d1bba5bf4c64914f300f122ec898"
format_stamp: "Formatted at 2018-06-19 07:11:43 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:43.609185 25059 fs_manager.cc:501] Time spent creating directory manager: real 0.014s	user 0.008s	sys 0.004s
I0619 00:11:43.610357 25059 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:11:43.611524 25059 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:11:43.619431 25059 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I0619 00:11:43.619557 25059 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/wal
uuid: "cb25d1bba5bf4c64914f300f122ec898"
format_stamp: "Formatted at 2018-06-19 07:11:43 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:43.619813 25059 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:11:43.798820 25059 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:11:43.801102 25059 master_main.cc:83] Starting Master server...
I0619 00:11:43.884034 25059 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.62:37587
I0619 00:11:43.884877 25059 webserver.cc:175] Starting webserver on 127.24.119.62:0
I0619 00:11:43.884975 25059 webserver.cc:186] Document root disabled
I0619 00:11:43.886795 25059 webserver.cc:313] Webserver started. Bound to: http://127.24.119.62:37703/
I0619 00:11:43.887773 25059 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-0/data/info.pb
I0619 00:11:43.890811 25120 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:43.892379 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 25059
I0619 00:11:43.893261 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:46325
--webserver_interface=127.24.119.61
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058 with env {}
I0619 00:11:43.902631 25120 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:11:43.924029 25120 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } has no permanent_uuid. Determining permanent_uuid...
W0619 00:11:43.925671 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:46325: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:43.945042 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } attempt: 1
W0619 00:11:43.946233 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:46325: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:43.994527 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } attempt: 2
W0619 00:11:43.995780 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:46325: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:11:44.054081 25121 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:11:44.054437 25121 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:11:44.063621 25121 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:11:44.063740 25121 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0619 00:11:44.063851 25121 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:11:44.063946 25121 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:11:44.064054 25121 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:11:44.083174 25121 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:46325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.119.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:11:44.091523 25121 master_main.cc:80] Initializing master server...
I0619 00:11:44.093813 25121 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 440413us
I0619 00:11:44.094585 25121 fs_manager.cc:261] Metadata directory not provided
I0619 00:11:44.094736 25121 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:11:44.094983 25121 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:11:44.095091 25121 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:11:44.104003 25121 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/instance:
uuid: "b2a016cc9f274bc1979068306ec6d15e"
format_stamp: "Formatted at 2018-06-19 07:11:44 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:44.104805 25121 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal/instance:
uuid: "b2a016cc9f274bc1979068306ec6d15e"
format_stamp: "Formatted at 2018-06-19 07:11:44 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:44.107107 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } attempt: 3
W0619 00:11:44.108395 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:46325: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:44.118911 25121 fs_manager.cc:501] Time spent creating directory manager: real 0.014s	user 0.009s	sys 0.004s
I0619 00:11:44.120076 25121 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:11:44.121275 25121 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:11:44.129240 25121 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I0619 00:11:44.129353 25121 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
uuid: "b2a016cc9f274bc1979068306ec6d15e"
format_stamp: "Formatted at 2018-06-19 07:11:44 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:44.129562 25121 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:11:44.284715 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } attempt: 4
W0619 00:11:44.285969 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:46325: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:44.323063 25121 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:11:44.325304 25121 master_main.cc:83] Starting Master server...
I0619 00:11:44.406949 25121 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.61:46325
I0619 00:11:44.407773 25121 webserver.cc:175] Starting webserver on 127.24.119.61:0
I0619 00:11:44.407871 25121 webserver.cc:186] Document root disabled
I0619 00:11:44.409708 25121 webserver.cc:313] Webserver started. Bound to: http://127.24.119.61:33317/
I0619 00:11:44.410405 25121 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/info.pb
I0619 00:11:44.413926 25184 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:44.419777 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 25121
I0619 00:11:44.420605 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.60:57058
--webserver_interface=127.24.119.60
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058 with env {}
I0619 00:11:44.426031 25184 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:11:44.442978 25184 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:11:44.455456 25184 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } has no permanent_uuid. Determining permanent_uuid...
W0619 00:11:44.457094 25184 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.476455 25184 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 1
W0619 00:11:44.477512 25184 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.525842 25184 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 2
W0619 00:11:44.527070 25184 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:11:44.584242 25185 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:11:44.584579 25185 flags.cc:406] Enabled unsafe flag: --never_fsync=true
I0619 00:11:44.589267 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } attempt: 5
W0619 00:11:44.595403 25185 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:11:44.595535 25185 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0619 00:11:44.595674 25185 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:11:44.595811 25185 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:11:44.595950 25185 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:11:44.595921 25120 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } has no permanent_uuid. Determining permanent_uuid...
W0619 00:11:44.597143 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.615123 25185 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.60:57058
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.119.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:11:44.623481 25185 master_main.cc:80] Initializing master server...
I0619 00:11:44.625262 25185 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 440413us
I0619 00:11:44.625986 25185 fs_manager.cc:261] Metadata directory not provided
I0619 00:11:44.626121 25185 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:11:44.626379 25185 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:11:44.626495 25185 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:11:44.635481 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 1
I0619 00:11:44.635643 25185 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data/instance:
uuid: "8c39479445934adb9bdbb6e60c3bf712"
format_stamp: "Formatted at 2018-06-19 07:11:44 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:44.636538 25185 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/wal/instance:
uuid: "8c39479445934adb9bdbb6e60c3bf712"
format_stamp: "Formatted at 2018-06-19 07:11:44 on quasar-xmwjxn-1.vpc.cloudera.com"
W0619 00:11:44.636744 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.638381 25184 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 3
W0619 00:11:44.639453 25184 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.650305 25185 fs_manager.cc:501] Time spent creating directory manager: real 0.013s	user 0.007s	sys 0.006s
I0619 00:11:44.651459 25185 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:11:44.652654 25185 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:11:44.660588 25185 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I0619 00:11:44.660742 25185 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/wal
uuid: "8c39479445934adb9bdbb6e60c3bf712"
format_stamp: "Formatted at 2018-06-19 07:11:44 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:44.660962 25185 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:11:44.685060 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 2
W0619 00:11:44.686539 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.786895 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 3
W0619 00:11:44.788662 25120 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.810123 25185 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:11:44.812439 25185 master_main.cc:83] Starting Master server...
I0619 00:11:44.815696 25184 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 4
W0619 00:11:44.816963 25184 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:57058: Network error: Client connection negotiation failed: client connection to 127.24.119.60:57058: connect: Connection refused (error 111)
I0619 00:11:44.897539 25185 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.60:57058
I0619 00:11:44.898481 25185 webserver.cc:175] Starting webserver on 127.24.119.60:0
I0619 00:11:44.898607 25185 webserver.cc:186] Document root disabled
I0619 00:11:44.900416 25185 webserver.cc:313] Webserver started. Bound to: http://127.24.119.60:57459/
I0619 00:11:44.901144 25185 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-2/data/info.pb
I0619 00:11:44.904517 25249 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:44.906153 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 25185
I0619 00:11:44.907212 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.24.119.1:0
--local_ip_for_outbound_sockets=127.24.119.1
--webserver_interface=127.24.119.1
--webserver_port=0
--tserver_master_addrs=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0619 00:11:44.916272 25249 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:11:44.928999 25120 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 4
I0619 00:11:44.932078 25249 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:11:44.942457 25249 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:11:44.945894 25120 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Bootstrap starting.
I0619 00:11:44.949060 25120 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: No blocks or log segments found. Creating new log.
I0619 00:11:44.950515 25120 log.cc:525] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Log is configured to *not* fsync() on all Append() calls
I0619 00:11:44.954910 25120 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: No bootstrap required, opened a new log
I0619 00:11:44.961884 25249 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Bootstrap starting.
I0619 00:11:44.965171 25249 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: No blocks or log segments found. Creating new log.
I0619 00:11:44.966601 25249 log.cc:525] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Log is configured to *not* fsync() on all Append() calls
I0619 00:11:44.971846 25249 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: No bootstrap required, opened a new log
I0619 00:11:44.972856 25120 raft_consensus.cc:325] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:44.973532 25120 raft_consensus.cc:351] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:44.973778 25120 raft_consensus.cc:628] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb25d1bba5bf4c64914f300f122ec898, State: Initialized, Role: FOLLOWER
I0619 00:11:44.974912 25120 consensus_queue.cc:227] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:44.979020 25120 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:44.979715 25120 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:44.980988 25120 sys_catalog.cc:424] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:11:44.988931 25249 raft_consensus.cc:325] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:44.989673 25249 raft_consensus.cc:351] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:44.989918 25249 raft_consensus.cc:628] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c39479445934adb9bdbb6e60c3bf712, State: Initialized, Role: FOLLOWER
I0619 00:11:44.990571 25249 consensus_queue.cc:227] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:44.998060 25257 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:44.999125 25257 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:44.979332 25255 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.081480 25255 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.081912 25255 raft_consensus.cc:457] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:45.082173 25255 raft_consensus.cc:2732] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Snoozing failure detection for 0.603s (starting election)
I0619 00:11:45.082512 25255 raft_consensus.cc:479] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:45.085276 25255 leader_election.cc:243] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325)
I0619 00:11:45.086961 25255 leader_election.cc:243] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:11:45.109341 25250 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:11:45.109968 25250 flags.cc:406] Enabled unsafe flag: --never_fsync=true
I0619 00:11:44.998849 25249 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.112975 25249 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.114574 25249 sys_catalog.cc:424] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:11:45.114708 25226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb25d1bba5bf4c64914f300f122ec898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712" is_pre_election: true
I0619 00:11:45.115499 25226 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 0.714s (vote granted)
I0619 00:11:45.115849 25226 raft_consensus.cc:2245] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb25d1bba5bf4c64914f300f122ec898 in term 0.
I0619 00:11:45.116976 25068 leader_election.cc:379] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:45.117286 25068 leader_election.cc:270] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:45.118404 25255 raft_consensus.cc:2732] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Snoozing failure detection for 0.572s (election complete)
I0619 00:11:45.118639 25255 raft_consensus.cc:2539] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:45.118880 25255 raft_consensus.cc:457] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:45.119076 25255 raft_consensus.cc:2732] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Snoozing failure detection for 0.529s (starting election)
I0619 00:11:45.119284 25255 raft_consensus.cc:2784] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:45.121765 25184 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } attempt: 5
I0619 00:11:45.125356 25255 raft_consensus.cc:479] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:45.126305 25185 master_main.cc:86] Master server successfully started.
I0619 00:11:45.126353 25255 leader_election.cc:243] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 election: Requesting vote from peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325)
I0619 00:11:45.126770 25255 leader_election.cc:243] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 election: Requesting vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:45.127876 25226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb25d1bba5bf4c64914f300f122ec898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712"
W0619 00:11:45.128123 25250 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
I0619 00:11:45.128314 25226 raft_consensus.cc:2784] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:45.128595 25226 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 0.673s (vote granted)
W0619 00:11:45.128828 25250 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.119.1
I0619 00:11:45.133816 25226 raft_consensus.cc:2245] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb25d1bba5bf4c64914f300f122ec898 in term 1.
I0619 00:11:45.134644 25068 leader_election.cc:379] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 election: Vote granted by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:45.134917 25068 leader_election.cc:270] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:45.138473 25184 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Bootstrap starting.
I0619 00:11:45.141525 25184 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: No blocks or log segments found. Creating new log.
I0619 00:11:45.141760 25255 raft_consensus.cc:2732] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Snoozing failure detection for 0.657s (election complete)
I0619 00:11:45.141996 25255 raft_consensus.cc:2539] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:45.142382 25255 raft_consensus.cc:593] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Becoming Leader. State: Replica: cb25d1bba5bf4c64914f300f122ec898, State: Running, Role: LEADER
I0619 00:11:45.142874 25255 consensus_queue.cc:204] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
W0619 00:11:45.144482 25259 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0619 00:11:45.144721 25259 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0619 00:11:45.147624 25184 log.cc:525] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Log is configured to *not* fsync() on all Append() calls
I0619 00:11:45.151908 25184 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: No bootstrap required, opened a new log
I0619 00:11:45.167125 25250 tablet_server_main.cc:78] Tablet server non-default flags:
--log_preallocate_segments=false
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.24.119.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.119.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.119.1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
W0619 00:11:45.054520 25076 tablet.cc:1678] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:11:45.180418 25059 master_main.cc:86] Master server successfully started.
I0619 00:11:45.184854 25258 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb25d1bba5bf4c64914f300f122ec898. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.185597 25258 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: This master's current role is: LEADER
I0619 00:11:45.187929 25250 tablet_server_main.cc:85] Initializing tablet server...
I0619 00:11:45.188773 25184 raft_consensus.cc:325] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:45.190847 25184 raft_consensus.cc:351] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:45.191247 25184 raft_consensus.cc:628] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a016cc9f274bc1979068306ec6d15e, State: Initialized, Role: FOLLOWER
I0619 00:11:45.192303 25184 consensus_queue.cc:227] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:45.194185 25250 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 440913us
I0619 00:11:45.196000 25250 fs_manager.cc:261] Metadata directory not provided
I0619 00:11:45.196279 25250 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:11:45.197852 25250 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:11:45.198263 25250 server_base.cc:433] Attempting to create new FS layout instead
W0619 00:11:45.200973 25256 catalog_manager.cc:1064] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0619 00:11:45.201210 25256 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0619 00:11:45.201483 25267 catalog_manager.cc:1006] Loading table and tablet metadata into memory...
I0619 00:11:45.198377 25184 sys_catalog.cc:337] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.202160 25184 sys_catalog.cc:340] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.203433 25184 sys_catalog.cc:424] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:11:45.212383 25267 catalog_manager.cc:1017] Initializing Kudu internal certificate authority...
I0619 00:11:45.226153 25250 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data/instance:
uuid: "c007a00e65ad41a79da1a0c1050c55b4"
format_stamp: "Formatted at 2018-06-19 07:11:45 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:45.234046 25250 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/instance:
uuid: "c007a00e65ad41a79da1a0c1050c55b4"
format_stamp: "Formatted at 2018-06-19 07:11:45 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:45.278956 25226 raft_consensus.cc:1081] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Refusing update from remote peer cb25d1bba5bf4c64914f300f122ec898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:45.280361 25255 consensus_queue.cc:963] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:45.289844 25250 fs_manager.cc:501] Time spent creating directory manager: real 0.055s	user 0.014s	sys 0.007s
I0619 00:11:45.291632 25250 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:11:45.293063 25250 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:11:45.310842 25257 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb25d1bba5bf4c64914f300f122ec898. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.311494 25257 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.311688 25250 fs_manager.cc:417] Time spent opening block manager: real 0.008s	user 0.001s	sys 0.005s
I0619 00:11:45.311897 25250 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal
uuid: "c007a00e65ad41a79da1a0c1050c55b4"
format_stamp: "Formatted at 2018-06-19 07:11:45 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:45.312255 25250 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:11:45.320108 25257 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.320778 25257 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.199048 25269 sys_catalog.cc:337] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.328586 25269 sys_catalog.cc:340] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.328279 25258 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.328963 25269 raft_consensus.cc:457] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:45.329025 25258 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: This master's current role is: LEADER
I0619 00:11:45.329208 25269 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.730s (starting election)
W0619 00:11:45.329974 25140 tablet.cc:1678] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:11:45.330433 25267 catalog_manager.cc:904] Generated new certificate authority record
I0619 00:11:45.329485 25269 raft_consensus.cc:479] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:45.332216 25269 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:45.332640 25267 catalog_manager.cc:1026] Loading token signing keys...
I0619 00:11:45.333287 25269 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:45.334174 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:45.334507 25226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712" is_pre_election: true
I0619 00:11:45.334869 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.335072 25226 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.336549 25133 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 1 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.337517 25132 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 1 pre-election: Vote denied by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058). Message: Invalid argument: T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.337779 25132 leader_election.cc:270] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:45.338728 25269 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.706s (election complete)
I0619 00:11:45.338969 25269 raft_consensus.cc:2784] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:45.344491 25269 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:45.348330 25161 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb25d1bba5bf4c64914f300f122ec898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a016cc9f274bc1979068306ec6d15e" is_pre_election: true
I0619 00:11:45.349522 25159 raft_consensus.cc:1081] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Refusing update from remote peer cb25d1bba5bf4c64914f300f122ec898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:45.355659 25121 master_main.cc:86] Master server successfully started.
I0619 00:11:45.355980 25161 raft_consensus.cc:2220] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb25d1bba5bf4c64914f300f122ec898 for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:11:45.357416 25071 leader_election.cc:393] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 pre-election: Vote denied by peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325). Message: Service unavailable: T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb25d1bba5bf4c64914f300f122ec898 for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:11:45.354640 25160 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb25d1bba5bf4c64914f300f122ec898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a016cc9f274bc1979068306ec6d15e"
I0619 00:11:45.359251 25255 consensus_queue.cc:963] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:45.359472 25160 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.803s (vote granted)
I0619 00:11:45.360213 25267 catalog_manager.cc:4090] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Generated new TSK 0
I0619 00:11:45.364838 25160 raft_consensus.cc:2245] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb25d1bba5bf4c64914f300f122ec898 in term 1.
I0619 00:11:45.366677 25071 leader_election.cc:379] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [CANDIDATE]: Term 1 election: Vote granted by peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325)
W0619 00:11:45.386581 25270 catalog_manager.cc:1064] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0619 00:11:45.392541 25270 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0619 00:11:45.424397 25269 sys_catalog.cc:337] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.425114 25269 sys_catalog.cc:340] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.425773 25271 sys_catalog.cc:337] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb25d1bba5bf4c64914f300f122ec898. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.426409 25271 sys_catalog.cc:340] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:45.428197 25258 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:45.428896 25258 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: This master's current role is: LEADER
I0619 00:11:45.626260 25250 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:11:45.629789 25250 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0619 00:11:45.632123 25250 tablet_server_main.cc:90] Starting tablet server...
I0619 00:11:45.879041 25376 raft_consensus.cc:457] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb25d1bba5bf4c64914f300f122ec898)
I0619 00:11:45.879480 25376 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 0.678s (starting election)
I0619 00:11:45.879809 25376 raft_consensus.cc:479] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:45.881016 25376 leader_election.cc:243] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:45.881585 25376 leader_election.cc:243] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325)
I0619 00:11:45.882445 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c39479445934adb9bdbb6e60c3bf712" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:45.882753 25160 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c39479445934adb9bdbb6e60c3bf712" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b2a016cc9f274bc1979068306ec6d15e" is_pre_election: true
I0619 00:11:45.883059 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.883321 25160 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.884179 25198 leader_election.cc:393] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.884863 25200 leader_election.cc:393] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325). Message: Invalid argument: T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.885114 25200 leader_election.cc:270] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:11:45.885619 25376 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 0.589s (election complete)
I0619 00:11:45.885841 25376 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:11:45.932195 25395 raft_consensus.cc:457] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb25d1bba5bf4c64914f300f122ec898)
I0619 00:11:45.932585 25395 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.743s (starting election)
I0619 00:11:45.932940 25395 raft_consensus.cc:479] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:45.933976 25395 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:45.934753 25395 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:45.935143 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:45.935688 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.935892 25226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712" is_pre_election: true
I0619 00:11:45.936342 25133 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.936447 25226 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.937181 25132 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058). Message: Invalid argument: T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:45.937429 25132 leader_election.cc:270] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:11:45.939750 25395 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.614s (election complete)
I0619 00:11:45.939990 25395 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:11:45.950372 25250 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.1:32951
I0619 00:11:45.951288 25250 webserver.cc:175] Starting webserver on 127.24.119.1:0
I0619 00:11:45.951395 25250 webserver.cc:186] Document root disabled
I0619 00:11:45.953536 25250 webserver.cc:313] Webserver started. Bound to: http://127.24.119.1:38474/
I0619 00:11:45.954216 25250 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/data/info.pb
I0619 00:11:45.958086 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 25250
I0619 00:11:45.959461 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data
--block_manager=log
--rpc_bind_addresses=127.24.119.2:0
--local_ip_for_outbound_sockets=127.24.119.2
--webserver_interface=127.24.119.2
--webserver_port=0
--tserver_master_addrs=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0619 00:11:46.005690 25250 tablet_server_main.cc:93] Tablet server successfully started.
I0619 00:11:46.027791 25401 heartbeater.cc:346] Connected to a master server at 127.24.119.62:37587
I0619 00:11:46.028108 25401 heartbeater.cc:426] Registering TS with master...
I0619 00:11:46.028729 25401 heartbeater.cc:475] Master 127.24.119.62:37587 requested a full tablet report, sending...
I0619 00:11:46.031195 25087 ts_manager.cc:81] Registered new tserver with Master: c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:46.033852 25087 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.24.119.1:38914
I0619 00:11:46.033991 25407 heartbeater.cc:346] Connected to a master server at 127.24.119.60:57058
I0619 00:11:46.034214 25407 heartbeater.cc:426] Registering TS with master...
I0619 00:11:46.035429 25407 heartbeater.cc:475] Master 127.24.119.60:57058 requested a full tablet report, sending...
I0619 00:11:46.040666 25216 ts_manager.cc:81] Registered new tserver with Master: c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:46.042639 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:46.042882 25402 heartbeater.cc:426] Registering TS with master...
I0619 00:11:46.043205 25402 heartbeater.cc:475] Master 127.24.119.61:46325 requested a full tablet report, sending...
I0619 00:11:46.045615 25151 ts_manager.cc:81] Registered new tserver with Master: c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:46.151695 25259 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: acquiring CA information for follower catalog manager: success
I0619 00:11:46.154569 25259 catalog_manager.cc:1090] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:11:46.156388 25403 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:11:46.156818 25403 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:11:46.165102 25403 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:11:46.165446 25403 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.119.2
I0619 00:11:46.187290 25403 tablet_server_main.cc:78] Tablet server non-default flags:
--log_preallocate_segments=false
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.24.119.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.119.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.119.2
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:11:46.196291 25403 tablet_server_main.cc:85] Initializing tablet server...
I0619 00:11:46.199196 25403 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 441413us
I0619 00:11:46.199656 25403 fs_manager.cc:261] Metadata directory not provided
I0619 00:11:46.199790 25403 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:11:46.200047 25403 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:11:46.200171 25403 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:11:46.209043 25403 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data/instance:
uuid: "10bda0b4856f4441b5aed24cdc4764a8"
format_stamp: "Formatted at 2018-06-19 07:11:46 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:46.209811 25403 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/instance:
uuid: "10bda0b4856f4441b5aed24cdc4764a8"
format_stamp: "Formatted at 2018-06-19 07:11:46 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:46.224833 25403 fs_manager.cc:501] Time spent creating directory manager: real 0.015s	user 0.007s	sys 0.007s
I0619 00:11:46.226055 25403 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:11:46.227223 25403 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:11:46.235810 25403 fs_manager.cc:417] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I0619 00:11:46.236017 25403 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal
uuid: "10bda0b4856f4441b5aed24cdc4764a8"
format_stamp: "Formatted at 2018-06-19 07:11:46 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:46.236274 25403 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:11:46.398553 25270 catalog_manager.cc:1062] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: acquiring CA information for follower catalog manager: success
I0619 00:11:46.401479 25270 catalog_manager.cc:1090] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0619 00:11:46.458391 25403 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:11:46.466711 25403 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0619 00:11:46.473120 25403 tablet_server_main.cc:90] Starting tablet server...
I0619 00:11:46.538159 25401 heartbeater.cc:467] Master 127.24.119.62:37587 was elected leader, sending a full tablet report...
I0619 00:11:46.549345 25436 raft_consensus.cc:457] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb25d1bba5bf4c64914f300f122ec898)
I0619 00:11:46.549809 25436 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.893s (starting election)
I0619 00:11:46.550112 25436 raft_consensus.cc:479] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:46.551236 25436 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:46.551697 25436 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:46.552592 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:46.552767 25226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712" is_pre_election: true
I0619 00:11:46.553145 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:46.553287 25226 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:46.553833 25133 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:46.554394 25132 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058). Message: Invalid argument: T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:46.554659 25132 leader_election.cc:270] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:11:46.555104 25436 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.509s (election complete)
I0619 00:11:46.555336 25436 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:11:47.142175 25515 raft_consensus.cc:457] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb25d1bba5bf4c64914f300f122ec898)
I0619 00:11:47.142470 25515 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.840s (starting election)
I0619 00:11:47.142799 25515 raft_consensus.cc:479] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:47.150197 25515 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:47.150635 25515 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:47.151687 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:47.151898 25226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712" is_pre_election: true
I0619 00:11:47.152330 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.152437 25226 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.153095 25133 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.153568 25132 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058). Message: Invalid argument: T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.153822 25132 leader_election.cc:270] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:11:47.154228 25515 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.865s (election complete)
I0619 00:11:47.154474 25515 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:11:47.289717 25403 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.2:38512
I0619 00:11:47.291043 25403 webserver.cc:175] Starting webserver on 127.24.119.2:0
I0619 00:11:47.291215 25403 webserver.cc:186] Document root disabled
I0619 00:11:47.297682 25403 webserver.cc:313] Webserver started. Bound to: http://127.24.119.2:60507/
I0619 00:11:47.298614 25403 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/data/info.pb
I0619 00:11:47.304353 25534 raft_consensus.cc:457] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb25d1bba5bf4c64914f300f122ec898)
I0619 00:11:47.304705 25534 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 0.978s (starting election)
I0619 00:11:47.305047 25534 raft_consensus.cc:479] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:47.305876 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 25403
I0619 00:11:47.306176 25534 leader_election.cc:243] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:47.306994 25534 leader_election.cc:243] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325)
I0619 00:11:47.307262 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.24.119.3:0
--local_ip_for_outbound_sockets=127.24.119.3
--webserver_interface=127.24.119.3
--webserver_port=0
--tserver_master_addrs=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0619 00:11:47.308234 25160 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c39479445934adb9bdbb6e60c3bf712" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b2a016cc9f274bc1979068306ec6d15e" is_pre_election: true
I0619 00:11:47.308821 25160 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.309648 25200 leader_election.cc:393] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325). Message: Invalid argument: T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.310125 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c39479445934adb9bdbb6e60c3bf712" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:47.310714 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.311401 25198 leader_election.cc:393] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.311686 25198 leader_election.cc:270] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:11:47.312368 25534 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 0.927s (election complete)
I0619 00:11:47.312646 25534 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:11:47.366647 25403 tablet_server_main.cc:93] Tablet server successfully started.
I0619 00:11:47.397229 25538 heartbeater.cc:346] Connected to a master server at 127.24.119.60:57058
I0619 00:11:47.397585 25538 heartbeater.cc:426] Registering TS with master...
I0619 00:11:47.398746 25538 heartbeater.cc:475] Master 127.24.119.60:57058 requested a full tablet report, sending...
I0619 00:11:47.400773 25535 heartbeater.cc:346] Connected to a master server at 127.24.119.62:37587
I0619 00:11:47.401002 25535 heartbeater.cc:426] Registering TS with master...
I0619 00:11:47.401751 25216 ts_manager.cc:81] Registered new tserver with Master: 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:47.401824 25535 heartbeater.cc:475] Master 127.24.119.62:37587 requested a full tablet report, sending...
I0619 00:11:47.403668 25087 ts_manager.cc:81] Registered new tserver with Master: 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:47.405393 25087 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.24.119.2:60771
I0619 00:11:47.411111 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:47.411340 25536 heartbeater.cc:426] Registering TS with master...
I0619 00:11:47.411695 25536 heartbeater.cc:475] Master 127.24.119.61:46325 requested a full tablet report, sending...
I0619 00:11:47.413758 25151 ts_manager.cc:81] Registered new tserver with Master: 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:11:47.609931 25537 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:11:47.610510 25537 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:11:47.627140 25537 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:11:47.627759 25537 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.119.3
I0619 00:11:47.661548 25537 tablet_server_main.cc:78] Tablet server non-default flags:
--log_preallocate_segments=false
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.24.119.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.119.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.119.3
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:11:47.677968 25537 tablet_server_main.cc:85] Initializing tablet server...
I0619 00:11:47.681329 25537 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 441913us
I0619 00:11:47.682068 25537 fs_manager.cc:261] Metadata directory not provided
I0619 00:11:47.682265 25537 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:11:47.682646 25537 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:11:47.682859 25537 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:11:47.698292 25537 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data/instance:
uuid: "395763cf325d4ce19859830a7ad35c85"
format_stamp: "Formatted at 2018-06-19 07:11:47 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:47.699446 25537 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/instance:
uuid: "395763cf325d4ce19859830a7ad35c85"
format_stamp: "Formatted at 2018-06-19 07:11:47 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:47.724160 25537 fs_manager.cc:501] Time spent creating directory manager: real 0.024s	user 0.013s	sys 0.008s
I0619 00:11:47.726161 25537 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:11:47.727566 25537 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:11:47.741763 25537 fs_manager.cc:417] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I0619 00:11:47.741979 25537 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal
uuid: "395763cf325d4ce19859830a7ad35c85"
format_stamp: "Formatted at 2018-06-19 07:11:47 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:47.742331 25537 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:11:47.864336 25557 raft_consensus.cc:457] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb25d1bba5bf4c64914f300f122ec898)
I0619 00:11:47.864684 25557 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 1.125s (starting election)
I0619 00:11:47.865042 25557 raft_consensus.cc:479] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:47.866093 25557 leader_election.cc:243] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:47.866529 25557 leader_election.cc:243] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325)
I0619 00:11:47.867305 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c39479445934adb9bdbb6e60c3bf712" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:47.867771 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.868062 25160 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c39479445934adb9bdbb6e60c3bf712" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b2a016cc9f274bc1979068306ec6d15e" is_pre_election: true
I0619 00:11:47.868465 25198 leader_election.cc:393] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.868646 25160 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.869287 25200 leader_election.cc:393] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325). Message: Invalid argument: T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c39479445934adb9bdbb6e60c3bf712 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:47.869540 25200 leader_election.cc:270] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:11:47.869977 25557 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712: Snoozing failure detection for 1.570s (election complete)
I0619 00:11:47.870219 25557 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:11:47.908500 25535 heartbeater.cc:467] Master 127.24.119.62:37587 was elected leader, sending a full tablet report...
I0619 00:11:47.931236 25537 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:11:47.934608 25537 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0619 00:11:47.936514 25537 tablet_server_main.cc:90] Starting tablet server...
I0619 00:11:48.125669 25537 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.3:39344
I0619 00:11:48.126518 25537 webserver.cc:175] Starting webserver on 127.24.119.3:0
I0619 00:11:48.126634 25537 webserver.cc:186] Document root disabled
I0619 00:11:48.128332 25537 webserver.cc:313] Webserver started. Bound to: http://127.24.119.3:53281/
I0619 00:11:48.129271 25537 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/data/info.pb
I0619 00:11:48.133388 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 25537
I0619 00:11:48.158504 25537 tablet_server_main.cc:93] Tablet server successfully started.
I0619 00:11:48.189482 25669 heartbeater.cc:346] Connected to a master server at 127.24.119.60:57058
I0619 00:11:48.189832 25669 heartbeater.cc:426] Registering TS with master...
I0619 00:11:48.190853 25669 heartbeater.cc:475] Master 127.24.119.60:57058 requested a full tablet report, sending...
I0619 00:11:48.193253 25216 ts_manager.cc:81] Registered new tserver with Master: 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:48.189487 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:48.199827 25668 heartbeater.cc:426] Registering TS with master...
I0619 00:11:48.200266 25668 heartbeater.cc:475] Master 127.24.119.61:46325 requested a full tablet report, sending...
I0619 00:11:48.195202 25667 heartbeater.cc:346] Connected to a master server at 127.24.119.62:37587
I0619 00:11:48.201347 25667 heartbeater.cc:426] Registering TS with master...
I0619 00:11:48.201649 25667 heartbeater.cc:475] Master 127.24.119.62:37587 requested a full tablet report, sending...
I0619 00:11:48.202476 25151 ts_manager.cc:81] Registered new tserver with Master: 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:48.203500 25087 ts_manager.cc:81] Registered new tserver with Master: 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:48.205072 25087 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.24.119.3:33192
I0619 00:11:48.208375 25052 external_mini_cluster.cc:490] 3 TS(s) registered with all masters
I0619 00:11:48.252110 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_6fc9e7dfe691413c9d9b134ccda69189"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:48.253464 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3c9df832d04346169a50a131227473d0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:48.255213 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3ee698b23f164542911f7dab51aba9e9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:48.260776 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_869875fa765c42d4ae2c9352322873f9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.270371 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3ee698b23f164542911f7dab51aba9e9 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:48.270371 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_869875fa765c42d4ae2c9352322873f9 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:48.270395 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3c9df832d04346169a50a131227473d0 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:48.271271 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6fc9e7dfe691413c9d9b134ccda69189 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.300674 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_869875fa765c42d4ae2c9352322873f9" } new_table_name: "default.table_bf9811157f1646299f1a2fa551667df3" alter_external_catalogs: true
I0619 00:11:48.328500 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 25121
I0619 00:11:48.332434 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_6fc9e7dfe691413c9d9b134ccda69189" }
I0619 00:11:48.332944 25086 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3c9df832d04346169a50a131227473d0" }
I0619 00:11:48.332913 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3ee698b23f164542911f7dab51aba9e9" } new_table_name: "default.table_7ea77d77ddd04410989cd2735133a5e5" alter_external_catalogs: true
I0619 00:11:48.342943 25084 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 19 ms (attempt = 1)
W0619 00:11:48.343236 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7507676a768e421dac27fe524538e0bb consensus state has no leader
I0619 00:11:48.350769 25087 catalog_manager.cc:4020] Not sending DeleteTablet requests; no consensus state for tablet fb784f543a5e4d1bbba8d11b8f8d06bd
I0619 00:11:48.352545 25086 catalog_manager.cc:4020] Not sending DeleteTablet requests; no consensus state for tablet 56ad7eacbbe34694acc3cbfc63a73892
I0619 00:11:48.353274 25085 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 48 ms (attempt = 1)
W0619 00:11:48.353559 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 has no consensus state
I0619 00:11:48.368194 25068 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 67 ms (attempt = 2)
W0619 00:11:48.368480 25068 catalog_manager.cc:2910] Async tablet task 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7507676a768e421dac27fe524538e0bb consensus state has no leader
I0619 00:11:48.386319 25603 tablet_service.cc:759] Processing CreateTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 (table=default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.387198 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6d6363f6403b4104b15b329ab7f0e483. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.387428 25604 tablet_service.cc:759] Processing CreateTablet for tablet 7507676a768e421dac27fe524538e0bb (table=default.table_869875fa765c42d4ae2c9352322873f9 [id=eeb1072dfe27492990409ab910065b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.388350 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7507676a768e421dac27fe524538e0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.400584 25470 tablet_service.cc:759] Processing CreateTablet for tablet 7507676a768e421dac27fe524538e0bb (table=default.table_869875fa765c42d4ae2c9352322873f9 [id=eeb1072dfe27492990409ab910065b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.401675 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7507676a768e421dac27fe524538e0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.403703 25336 tablet_service.cc:759] Processing CreateTablet for tablet 7507676a768e421dac27fe524538e0bb (table=default.table_869875fa765c42d4ae2c9352322873f9 [id=eeb1072dfe27492990409ab910065b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.404539 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7507676a768e421dac27fe524538e0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.415757 25604 ts_tablet_manager.cc:1173] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:48.418274 25071 connection.cc:512] client connection to 127.24.119.61:46325 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:11:48.418552 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:48.420217 25603 ts_tablet_manager.cc:1173] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:48.420306 25558 connection.cc:512] client connection to 127.24.119.61:46325 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.61:46325): Connection reset by peer (error 104)
I0619 00:11:48.425212 25724 ts_tablet_manager.cc:1028] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.425758 25724 tablet_bootstrap.cc:427] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:48.429986 25724 tablet_bootstrap.cc:578] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.430402 25071 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 33 ms (attempt = 2)
W0619 00:11:48.430702 25071 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.431829 25470 ts_tablet_manager.cc:1173] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.431985 25724 log.cc:525] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Log is configured to *not* fsync() on all Append() calls
I0619 00:11:48.436718 25068 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 71 ms (attempt = 3)
W0619 00:11:48.436983 25068 catalog_manager.cc:2910] Async tablet task 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7507676a768e421dac27fe524538e0bb consensus state has no leader
I0619 00:11:48.441938 25724 tablet_bootstrap.cc:427] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.442312 25724 ts_tablet_manager.cc:1045] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.006s
I0619 00:11:48.443348 25336 ts_tablet_manager.cc:1173] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.458796 25724 raft_consensus.cc:325] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } }
I0619 00:11:48.459755 25724 raft_consensus.cc:351] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.459973 25724 raft_consensus.cc:628] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:48.460777 25724 consensus_queue.cc:227] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } }
I0619 00:11:48.464452 25071 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 112 ms (attempt = 3)
W0619 00:11:48.464749 25071 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.468760 25667 heartbeater.cc:467] Master 127.24.119.62:37587 was elected leader, sending a full tablet report...
I0619 00:11:48.471017 25086 master_service.cc:523] ReplaceTablet: received request to replace tablet 7507676a768e421dac27fe524538e0bb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:48.473537 25724 ts_tablet_manager.cc:1073] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.031s	user 0.018s	sys 0.004s
I0619 00:11:48.473883 25724 ts_tablet_manager.cc:1028] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
W0619 00:11:48.475476 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.476991 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 7507676a768e421dac27fe524538e0bb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:48.481137 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 7507676a768e421dac27fe524538e0bb
I0619 00:11:48.481465 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:48.481899 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:48.482244 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:48.482553 25086 catalog_manager.cc:4596] ReplaceTablet: tablet 7507676a768e421dac27fe524538e0bb deleted and replaced by tablet baf80fd968c44e679987eb4ebe4473de
I0619 00:11:48.483929 25604 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.484061 25085 catalog_manager.cc:3663] Got report from deleted tablet 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) (replaced by tablet baf80fd968c44e679987eb4ebe4473de): Sending delete request for this tablet
I0619 00:11:48.484521 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]). Expected version 1 got 0
I0619 00:11:48.484740 25724 tablet_bootstrap.cc:427] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:48.487217 25724 tablet_bootstrap.cc:578] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.487202 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:48.487642 25085 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 65 ms (attempt = 1)
W0619 00:11:48.487902 25085 catalog_manager.cc:3945] Failed to send 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.489560 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.496284 25603 tablet_service.cc:759] Processing CreateTablet for tablet baf80fd968c44e679987eb4ebe4473de (table=default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.496857 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet baf80fd968c44e679987eb4ebe4473de. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.500533 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d946420480b24fe2810193b44fafa15b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.501897 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d946420480b24fe2810193b44fafa15b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.503767 25730 tablet_replica.cc:261] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:48.508638 25068 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 139 ms (attempt = 4)
W0619 00:11:48.508963 25068 catalog_manager.cc:2910] Async tablet task 7507676a768e421dac27fe524538e0bb (table default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7507676a768e421dac27fe524538e0bb consensus state has no leader
I0619 00:11:48.508785 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_14eb07c70b3f485b87fdb436a2ab7c27"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.509683 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_14eb07c70b3f485b87fdb436a2ab7c27 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.515429 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_4a989339185d4e3fa0c743cabbcbf619"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.516239 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4a989339185d4e3fa0c743cabbcbf619 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.518930 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_706cbf40bb854d3ebbdbdbf69f4e15d6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.519783 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:48.504228 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.507335 25603 ts_tablet_manager.cc:1173] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.542162 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.525336 25604 tablet_service.cc:759] Processing CreateTablet for tablet 985e23a32fa84104a06dd19d97117ab2 (table=default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.542604 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.542852 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.543071 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.543315 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.543505 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.543725 25603 tablet_service.cc:759] Processing CreateTablet for tablet e3708e05f6054306b5bf7f1df2140e3d (table=default.table_14eb07c70b3f485b87fdb436a2ab7c27 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.551034 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.542809 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 985e23a32fa84104a06dd19d97117ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.551908 25724 tablet_bootstrap.cc:427] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.551973 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e3708e05f6054306b5bf7f1df2140e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.552305 25724 ts_tablet_manager.cc:1045] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.068s	user 0.006s	sys 0.001s
I0619 00:11:48.554026 25602 tablet_service.cc:759] Processing CreateTablet for tablet dc04126054714196af6bdf7e771ffab0 (table=default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.553807 25724 raft_consensus.cc:325] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.554504 25724 raft_consensus.cc:351] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.554731 25724 raft_consensus.cc:628] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:48.554844 25069 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 38 ms (attempt = 2)
W0619 00:11:48.555491 25069 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.555241 25724 consensus_queue.cc:227] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.555601 25601 tablet_service.cc:759] Processing CreateTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 (table=default.table_4a989339185d4e3fa0c743cabbcbf619 [id=f25f5cc3e1d64c95a0dcc4e36416c416]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.558606 25724 ts_tablet_manager.cc:1073] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
W0619 00:11:48.558935 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.559171 25730 raft_consensus.cc:2042] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:48.559505 25730 raft_consensus.cc:2069] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:48.561637 25730 ts_tablet_manager.cc:1369] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:48.561746 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.558943 25724 ts_tablet_manager.cc:1028] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.562964 25724 tablet_bootstrap.cc:427] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:48.563308 25086 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]). Expected version 1 got 0
I0619 00:11:48.564040 25086 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 50 ms (attempt = 1)
W0619 00:11:48.564296 25086 catalog_manager.cc:3945] Failed to send 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.567018 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dc04126054714196af6bdf7e771ffab0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.567878 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 89a7ff9cdbc046779799e4e8f71adb42. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.572604 25603 ts_tablet_manager.cc:1173] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.574173 25086 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]). Expected version 1 got 0
I0619 00:11:48.575284 25086 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 41 ms (attempt = 1)
W0619 00:11:48.575525 25086 catalog_manager.cc:3945] Failed to send 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.577670 25071 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 156 ms (attempt = 4)
W0619 00:11:48.577916 25071 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.580577 25604 ts_tablet_manager.cc:1173] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.584470 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_4a989339185d4e3fa0c743cabbcbf619" } new_table_name: "default.table_76fe64ca5bbd44279ae948d8df882d65" alter_external_catalogs: true
I0619 00:11:48.588933 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_d946420480b24fe2810193b44fafa15b" }
I0619 00:11:48.589570 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_bf9811157f1646299f1a2fa551667df3" } new_table_name: "default.table_5e925f129a1043b7ab80c112a0ee33e0" alter_external_catalogs: true
I0619 00:11:48.591184 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_7ea77d77ddd04410989cd2735133a5e5" }
I0619 00:11:48.593235 25602 ts_tablet_manager.cc:1173] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.594753 25069 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC with a delay of 111 ms (attempt = 3)
W0619 00:11:48.595018 25069 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.596469 25086 catalog_manager.cc:2891] Scheduling retry of 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC with a delay of 29 ms (attempt = 1)
W0619 00:11:48.596763 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89a7ff9cdbc046779799e4e8f71adb42 consensus state has no leader
I0619 00:11:48.599524 25601 ts_tablet_manager.cc:1173] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.602423 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 985e23a32fa84104a06dd19d97117ab2
I0619 00:11:48.602736 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.603394 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.603790 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.605924 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.607520 25084 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 31 ms (attempt = 1)
W0619 00:11:48.607815 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.606765 25724 tablet_bootstrap.cc:578] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.612655 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:48.612931 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.613289 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.613680 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
W0619 00:11:48.615836 25071 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.616430 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:48.618058 25067 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.619896 25730 ts_tablet_manager.cc:1382] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:48.620151 25730 log.cc:986] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/7507676a768e421dac27fe524538e0bb
I0619 00:11:48.623492 25730 ts_tablet_manager.cc:1403] T 7507676a768e421dac27fe524538e0bb P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:48.626673 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) successfully deleted
I0619 00:11:48.626700 25068 catalog_manager.cc:2891] Scheduling retry of 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC with a delay of 63 ms (attempt = 2)
W0619 00:11:48.627005 25068 catalog_manager.cc:2910] Async tablet task 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 89a7ff9cdbc046779799e4e8f71adb42 consensus state has no leader
I0619 00:11:48.627240 25724 tablet_bootstrap.cc:427] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.627542 25724 ts_tablet_manager.cc:1045] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.065s	user 0.004s	sys 0.003s
W0619 00:11:48.628566 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 7507676a768e421dac27fe524538e0bb because the tablet was not found. No further retry: Not found: Tablet not found: 7507676a768e421dac27fe524538e0bb
I0619 00:11:48.628877 25730 tablet_replica.cc:261] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
W0619 00:11:48.629045 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 985e23a32fa84104a06dd19d97117ab2 already in progress: creating tablet
I0619 00:11:48.629115 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:48.629267 25071 catalog_manager.cc:2891] Scheduling retry of 985e23a32fa84104a06dd19d97117ab2 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 30 ms (attempt = 1)
I0619 00:11:48.629302 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:48.629472 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:48.629683 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:48.629890 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:48.630075 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:48.630267 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:48.630466 25730 raft_consensus.cc:2042] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:48.630123 25724 raft_consensus.cc:325] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.630965 25724 raft_consensus.cc:351] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.631145 25730 raft_consensus.cc:2069] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:48.631167 25724 raft_consensus.cc:628] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:48.632095 25724 consensus_queue.cc:227] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:48.634663 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.635473 25724 ts_tablet_manager.cc:1073] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I0619 00:11:48.635846 25724 ts_tablet_manager.cc:1028] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.636204 25724 tablet_bootstrap.cc:427] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:48.636471 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]). Expected version 2 got 0
I0619 00:11:48.637241 25087 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 51 ms (attempt = 1)
W0619 00:11:48.637490 25087 catalog_manager.cc:3945] Failed to send baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
W0619 00:11:48.638814 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.639375 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 32 ms (attempt = 2)
W0619 00:11:48.639652 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.641047 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]). Expected version 2 got 0
I0619 00:11:48.641752 25087 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 63 ms (attempt = 1)
W0619 00:11:48.642102 25087 catalog_manager.cc:3945] Failed to send baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.642225 25724 tablet_bootstrap.cc:578] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.642477 25730 ts_tablet_manager.cc:1369] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:48.647416 25729 raft_consensus.cc:457] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:48.647703 25729 raft_consensus.cc:2732] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.848s (starting election)
I0619 00:11:48.647979 25729 raft_consensus.cc:479] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.649998 25729 leader_election.cc:243] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:48.650920 25068 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 285 ms (attempt = 5)
W0619 00:11:48.651168 25068 catalog_manager.cc:2910] Async tablet task 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7507676a768e421dac27fe524538e0bb consensus state has no leader
I0619 00:11:48.650516 25729 leader_election.cc:243] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:48.655040 25676 tablet.cc:1678] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:11:48.656705 25724 tablet_bootstrap.cc:427] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.657035 25724 ts_tablet_manager.cc:1045] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.005s
I0619 00:11:48.658382 25724 raft_consensus.cc:325] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.659037 25724 raft_consensus.cc:351] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.659242 25724 raft_consensus.cc:628] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:48.660006 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.659806 25724 consensus_queue.cc:227] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.661006 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:48.663131 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:48.666913 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.669387 25730 ts_tablet_manager.cc:1382] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:48.669642 25730 log.cc:986] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:48.670099 25730 ts_tablet_manager.cc:1403] T 6d6363f6403b4104b15b329ab7f0e483 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:48.672453 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 113 ms (attempt = 3)
W0619 00:11:48.672749 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.673919 25724 ts_tablet_manager.cc:1073] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.000s
I0619 00:11:48.674330 25724 ts_tablet_manager.cc:1028] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.674701 25724 tablet_bootstrap.cc:427] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:48.676192 25553 leader_election.cc:306] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Not found: Tablet not found: baf80fd968c44e679987eb4ebe4473de
I0619 00:11:48.678112 25724 tablet_bootstrap.cc:578] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.679949 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) successfully deleted
W0619 00:11:48.680793 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 985e23a32fa84104a06dd19d97117ab2 already in progress: creating tablet
I0619 00:11:48.681041 25071 catalog_manager.cc:2891] Scheduling retry of 985e23a32fa84104a06dd19d97117ab2 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 45 ms (attempt = 2)
I0619 00:11:48.404873 25469 tablet_service.cc:759] Processing CreateTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 (table=default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.682332 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6d6363f6403b4104b15b329ab7f0e483. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.686691 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 7507676a768e421dac27fe524538e0bb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:48.689064 25724 tablet_bootstrap.cc:427] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.689642 25068 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 61 ms (attempt = 2)
W0619 00:11:48.689950 25068 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.689368 25724 ts_tablet_manager.cc:1045] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.006s
I0619 00:11:48.690179 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 7507676a768e421dac27fe524538e0bb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:48.691470 25068 catalog_manager.cc:2891] Scheduling retry of 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC with a delay of 98 ms (attempt = 3)
W0619 00:11:48.691731 25068 catalog_manager.cc:2910] Async tablet task 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 89a7ff9cdbc046779799e4e8f71adb42 consensus state has no leader
I0619 00:11:48.692083 25724 raft_consensus.cc:325] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.692654 25724 raft_consensus.cc:351] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.692872 25724 raft_consensus.cc:628] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:48.693354 25724 consensus_queue.cc:227] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.695093 25469 ts_tablet_manager.cc:1173] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:48.698128 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.699481 25087 catalog_manager.cc:3663] Got report from deleted tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:48.700212 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
W0619 00:11:48.700515 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.701784 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:48.703191 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 985e23a32fa84104a06dd19d97117ab2 already in progress: creating tablet
I0619 00:11:48.703415 25071 catalog_manager.cc:2891] Scheduling retry of 985e23a32fa84104a06dd19d97117ab2 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 32 ms (attempt = 1)
I0619 00:11:48.704294 25087 catalog_manager.cc:3663] Got report from deleted tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:48.704640 25724 ts_tablet_manager.cc:1073] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I0619 00:11:48.704937 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.704991 25724 ts_tablet_manager.cc:1028] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.705688 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 46 ms (attempt = 2)
W0619 00:11:48.705957 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.706813 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.705325 25724 tablet_bootstrap.cc:427] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:48.707226 25730 tablet_replica.cc:261] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:48.707451 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(985e23a32fa84104a06dd19d97117ab2)
W0619 00:11:48.707571 25069 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.707666 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:48.707844 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:48.708030 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:48.708238 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:48.708437 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:48.708631 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:48.708813 25730 raft_consensus.cc:2042] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:48.709066 25730 raft_consensus.cc:2069] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:48.420666 25423 connection.cc:512] client connection to 127.24.119.61:46325 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:11:48.710373 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:11:48.710686 25423 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
I0619 00:11:48.710853 25724 tablet_bootstrap.cc:578] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.711189 25730 ts_tablet_manager.cc:1369] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:48.714381 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.716295 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_de92feb7880c4ea2b3fd76c65c6b3df5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.717232 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_de92feb7880c4ea2b3fd76c65c6b3df5 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.718295 25724 tablet_bootstrap.cc:427] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.718830 25724 ts_tablet_manager.cc:1045] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.006s
I0619 00:11:48.714694 25465 tablet_service.cc:759] Processing CreateTablet for tablet dc04126054714196af6bdf7e771ffab0 (table=default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.720132 25724 raft_consensus.cc:325] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.721489 25724 raft_consensus.cc:351] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.721731 25724 raft_consensus.cc:628] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:48.722281 25724 consensus_queue.cc:227] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.726804 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.728250 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:48.733119 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.733608 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_77bc6b25b36d4879ba67ea600dcc9114"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:48.734220 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:48.734510 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_77bc6b25b36d4879ba67ea600dcc9114 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:48.735082 25071 catalog_manager.cc:2910] Async tablet task 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6363f6403b4104b15b329ab7f0e483 consensus state has no leader
I0619 00:11:48.734519 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_e2e6deb5e5764586802390a904ba068f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.736359 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e2e6deb5e5764586802390a904ba068f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.720927 25468 tablet_service.cc:759] Processing CreateTablet for tablet baf80fd968c44e679987eb4ebe4473de (table=default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.737813 25465 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dc04126054714196af6bdf7e771ffab0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.740058 25730 ts_tablet_manager.cc:1382] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:48.740293 25730 log.cc:986] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/985e23a32fa84104a06dd19d97117ab2
I0619 00:11:48.734127 25467 tablet_service.cc:759] Processing CreateTablet for tablet 985e23a32fa84104a06dd19d97117ab2 (table=default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.742686 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet baf80fd968c44e679987eb4ebe4473de. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.742936 25464 tablet_service.cc:759] Processing CreateTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 (table=default.table_4a989339185d4e3fa0c743cabbcbf619 [id=f25f5cc3e1d64c95a0dcc4e36416c416]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.743324 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.735198 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.745108 25730 ts_tablet_manager.cc:1403] T 985e23a32fa84104a06dd19d97117ab2 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:48.744992 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_9bbf1c0d460e4aafa3551e40c25513c4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.745841 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9bbf1c0d460e4aafa3551e40c25513c4 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.747282 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) successfully deleted
W0619 00:11:48.747805 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 because the tablet was not found. No further retry: Not found: Tablet not found: 985e23a32fa84104a06dd19d97117ab2
I0619 00:11:48.748594 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.743350 25466 tablet_service.cc:759] Processing CreateTablet for tablet e3708e05f6054306b5bf7f1df2140e3d (table=default.table_14eb07c70b3f485b87fdb436a2ab7c27 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.753623 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 99 ms (attempt = 3)
W0619 00:11:48.753906 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.755266 25068 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 89 ms (attempt = 3)
W0619 00:11:48.755530 25068 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.755802 25725 ts_tablet_manager.cc:1028] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:48.756364 25725 tablet_bootstrap.cc:427] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:48.758792 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 because the tablet was not found. No further retry: Not found: Tablet not found: 985e23a32fa84104a06dd19d97117ab2
I0619 00:11:48.760682 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 985e23a32fa84104a06dd19d97117ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.762090 25466 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e3708e05f6054306b5bf7f1df2140e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.761500 25725 tablet_bootstrap.cc:578] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:48.764405 25558 leader_election.cc:306] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: baf80fd968c44e679987eb4ebe4473de
I0619 00:11:48.764684 25558 leader_election.cc:270] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:48.764747 25470 tablet_service.cc:759] Processing CreateTablet for tablet a226a7e97a7e44af8301ed4e6363effb (table=default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.764748 25601 tablet_service.cc:759] Processing CreateTablet for tablet a226a7e97a7e44af8301ed4e6363effb (table=default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.765321 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a226a7e97a7e44af8301ed4e6363effb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.766916 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 89a7ff9cdbc046779799e4e8f71adb42. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.767786 25729 raft_consensus.cc:2732] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.890s (election complete)
I0619 00:11:48.768054 25729 raft_consensus.cc:2487] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:48.775527 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.776130 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a226a7e97a7e44af8301ed4e6363effb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.785179 25724 ts_tablet_manager.cc:1073] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.066s	user 0.004s	sys 0.000s
I0619 00:11:48.785606 25724 ts_tablet_manager.cc:1028] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.786043 25724 tablet_bootstrap.cc:427] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:48.787819 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 143 ms (attempt = 4)
W0619 00:11:48.788094 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.788043 25602 tablet_service.cc:759] Processing CreateTablet for tablet ac129ed44f68432f800420175aa0a630 (table=default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.788044 25604 tablet_service.cc:759] Processing CreateTablet for tablet fb8e94875a8347a0a473c95a77c38795 (table=default.table_77bc6b25b36d4879ba67ea600dcc9114 [id=41f48f0e1f87464eb43c5bf2a36f70ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.788650 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ac129ed44f68432f800420175aa0a630. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.789005 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fb8e94875a8347a0a473c95a77c38795. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.788877 25469 tablet_service.cc:759] Processing CreateTablet for tablet ac129ed44f68432f800420175aa0a630 (table=default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.789343 25725 log.cc:525] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Log is configured to *not* fsync() on all Append() calls
I0619 00:11:48.789458 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ac129ed44f68432f800420175aa0a630. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.792935 25068 catalog_manager.cc:2891] Scheduling retry of 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC with a delay of 164 ms (attempt = 4)
W0619 00:11:48.793254 25068 catalog_manager.cc:2910] Async tablet task 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 89a7ff9cdbc046779799e4e8f71adb42 consensus state has no leader
I0619 00:11:48.800945 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_14eb07c70b3f485b87fdb436a2ab7c27" } new_table_name: "default.table_514667669c72462a95c41b6bb68dc0fe" alter_external_catalogs: true
I0619 00:11:48.803503 25601 ts_tablet_manager.cc:1173] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.807723 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_706cbf40bb854d3ebbdbdbf69f4e15d6" }
I0619 00:11:48.809146 25603 tablet_service.cc:759] Processing CreateTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table=default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.810303 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_77bc6b25b36d4879ba67ea600dcc9114" } new_table_name: "default.table_c297d9ff075944df87aac8f56d2f9d95" alter_external_catalogs: true
I0619 00:11:48.814085 25086 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 46 ms (attempt = 1)
W0619 00:11:48.814363 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:48.816860 25086 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_76fe64ca5bbd44279ae948d8df882d65" }
I0619 00:11:48.818753 25724 tablet_bootstrap.cc:578] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.805541 25462 tablet_service.cc:759] Processing CreateTablet for tablet fb8e94875a8347a0a473c95a77c38795 (table=default.table_77bc6b25b36d4879ba67ea600dcc9114 [id=41f48f0e1f87464eb43c5bf2a36f70ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.816035 25470 ts_tablet_manager.cc:1173] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:48.823631 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7507676a768e421dac27fe524538e0bb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7507676a768e421dac27fe524538e0bb already in progress: creating tablet
I0619 00:11:48.823899 25071 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
I0619 00:11:48.824322 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet dc04126054714196af6bdf7e771ffab0
I0619 00:11:48.824774 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fb8e94875a8347a0a473c95a77c38795. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.824636 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.825495 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.825553 25466 ts_tablet_manager.cc:1173] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.826225 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.829780 25601 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.830162 25604 ts_tablet_manager.cc:1173] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.830498 25730 tablet_replica.cc:261] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:48.830726 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:48.830833 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc22ad98ed2e4e3eb2c62360626a6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.830911 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:48.831162 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:48.831349 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:48.831562 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:48.831748 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:48.831825 25465 ts_tablet_manager.cc:1173] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.831918 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:48.832085 25730 raft_consensus.cc:2042] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:48.832326 25730 raft_consensus.cc:2069] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:48.833722 25730 ts_tablet_manager.cc:1369] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:48.826707 25463 tablet_service.cc:759] Processing CreateTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table=default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:48.835939 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 because the tablet was not found. No further retry: Not found: Tablet not found: 985e23a32fa84104a06dd19d97117ab2
W0619 00:11:48.836401 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6d6363f6403b4104b15b329ab7f0e483 already in progress: creating tablet
I0619 00:11:48.836637 25071 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 32 ms (attempt = 1)
I0619 00:11:48.836889 25602 ts_tablet_manager.cc:1173] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.827069 25470 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.840173 25603 ts_tablet_manager.cc:1173] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.843413 25084 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 45 ms (attempt = 1)
W0619 00:11:48.843711 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:48.843706 25464 ts_tablet_manager.cc:1173] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.844249 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc22ad98ed2e4e3eb2c62360626a6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.847883 25068 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 130 ms (attempt = 4)
W0619 00:11:48.848153 25068 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.850934 25468 ts_tablet_manager.cc:1173] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.851330 25467 ts_tablet_manager.cc:1173] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.852953 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:48.853238 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.853641 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.854005 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.854842 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 137 ms (attempt = 4)
W0619 00:11:48.855098 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.856456 25604 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.856577 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.416308 25335 tablet_service.cc:759] Processing CreateTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 (table=default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.857784 25335 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6d6363f6403b4104b15b329ab7f0e483. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.860935 25469 ts_tablet_manager.cc:1173] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.861848 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 48 ms (attempt = 2)
W0619 00:11:48.862126 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:48.864406 25730 ts_tablet_manager.cc:1382] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:48.864640 25730 log.cc:986] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/dc04126054714196af6bdf7e771ffab0
I0619 00:11:48.865130 25730 ts_tablet_manager.cc:1403] T dc04126054714196af6bdf7e771ffab0 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:48.869168 25463 ts_tablet_manager.cc:1173] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.869964 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.870512 25724 tablet_bootstrap.cc:427] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.870841 25724 ts_tablet_manager.cc:1045] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.085s	user 0.004s	sys 0.002s
I0619 00:11:48.871206 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.872297 25335 ts_tablet_manager.cc:1173] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:48.872368 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet dc04126054714196af6bdf7e771ffab0 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet dc04126054714196af6bdf7e771ffab0 already in progress: creating tablet
I0619 00:11:48.872613 25071 catalog_manager.cc:2891] Scheduling retry of dc04126054714196af6bdf7e771ffab0 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 31 ms (attempt = 1)
I0619 00:11:48.872197 25724 raft_consensus.cc:325] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.872848 25724 raft_consensus.cc:351] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.873034 25724 raft_consensus.cc:628] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:48.873351 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 89a7ff9cdbc046779799e4e8f71adb42 already in progress: creating tablet
I0619 00:11:48.873558 25071 catalog_manager.cc:2891] Scheduling retry of 89a7ff9cdbc046779799e4e8f71adb42 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 61 ms (attempt = 1)
I0619 00:11:48.873520 25724 consensus_queue.cc:227] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:48.873942 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6d6363f6403b4104b15b329ab7f0e483 already in progress: creating tablet
I0619 00:11:48.874155 25071 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 65 ms (attempt = 2)
W0619 00:11:48.876096 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.876137 25724 ts_tablet_manager.cc:1073] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I0619 00:11:48.876510 25724 ts_tablet_manager.cc:1028] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.876863 25724 tablet_bootstrap.cc:427] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:48.878412 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.878739 25086 catalog_manager.cc:3663] Got report from deleted tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:48.879237 25462 ts_tablet_manager.cc:1173] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:48.879412 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.880484 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet dc04126054714196af6bdf7e771ffab0 (table default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]) successfully deleted
I0619 00:11:48.881733 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.882637 25730 tablet_replica.cc:261] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:48.882964 25086 catalog_manager.cc:3663] Got report from deleted tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:48.883505 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:48.883500 25725 tablet_bootstrap.cc:427] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:48.883601 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.883934 25725 ts_tablet_manager.cc:1045] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.128s	user 0.008s	sys 0.016s
I0619 00:11:48.884516 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:48.885480 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:48.885913 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.886240 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:48.886445 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:48.886629 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:48.886870 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(89a7ff9cdbc046779799e4e8f71adb42)
W0619 00:11:48.887217 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7507676a768e421dac27fe524538e0bb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7507676a768e421dac27fe524538e0bb already in progress: creating tablet
I0619 00:11:48.887300 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:48.887429 25071 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 60 ms (attempt = 2)
I0619 00:11:48.887513 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:48.887766 25730 raft_consensus.cc:2042] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:48.888041 25730 raft_consensus.cc:2069] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:48.890238 25068 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 58 ms (attempt = 2)
W0619 00:11:48.890509 25068 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:48.889941 25730 ts_tablet_manager.cc:1369] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:48.892087 25086 master_service.cc:523] ReplaceTablet: received request to replace tablet 6d6363f6403b4104b15b329ab7f0e483 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:48.895778 25086 master_service.cc:523] ReplaceTablet: received request to replace tablet 7507676a768e421dac27fe524538e0bb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:48.896070 25724 tablet_bootstrap.cc:578] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:48.417999 25290 connection.cc:512] client connection to 127.24.119.61:46325 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.61:46325): Connection reset by peer (error 104)
I0619 00:11:48.903911 25727 ts_tablet_manager.cc:1028] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:48.905467 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.905531 25727 tablet_bootstrap.cc:427] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:48.906747 25462 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.905485 25334 tablet_service.cc:759] Processing CreateTablet for tablet baf80fd968c44e679987eb4ebe4473de (table=default.table_bf9811157f1646299f1a2fa551667df3 [id=eeb1072dfe27492990409ab910065b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.906167 25335 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:48.907752 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet dc04126054714196af6bdf7e771ffab0 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet dc04126054714196af6bdf7e771ffab0 already in progress: creating tablet
I0619 00:11:48.907971 25071 catalog_manager.cc:2891] Scheduling retry of dc04126054714196af6bdf7e771ffab0 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 79 ms (attempt = 2)
I0619 00:11:48.907258 25332 tablet_service.cc:759] Processing CreateTablet for tablet e3708e05f6054306b5bf7f1df2140e3d (table=default.table_14eb07c70b3f485b87fdb436a2ab7c27 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.909307 25328 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:48.910181 25333 tablet_service.cc:759] Processing CreateTablet for tablet 985e23a32fa84104a06dd19d97117ab2 (table=default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.910882 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 77 ms (attempt = 3)
W0619 00:11:48.911161 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:48.909299 25329 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:48.909448 25334 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet baf80fd968c44e679987eb4ebe4473de. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.912942 25724 tablet_bootstrap.cc:427] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:48.913256 25724 ts_tablet_manager.cc:1045] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.036s	user 0.005s	sys 0.002s
I0619 00:11:48.914613 25724 raft_consensus.cc:325] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:48.915225 25724 raft_consensus.cc:351] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:48.915411 25724 raft_consensus.cc:628] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:48.918334 25724 consensus_queue.cc:227] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:48.921455 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:48.923209 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.919785 25333 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 985e23a32fa84104a06dd19d97117ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.924366 25324 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:48.925118 25724 ts_tablet_manager.cc:1073] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.001s
I0619 00:11:48.925732 25724 ts_tablet_manager.cc:1028] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:48.926106 25724 tablet_bootstrap.cc:427] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:48.925334 25730 ts_tablet_manager.cc:1382] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:48.927767 25730 log.cc:986] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:48.928329 25730 ts_tablet_manager.cc:1403] T 89a7ff9cdbc046779799e4e8f71adb42 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:48.930003 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) successfully deleted
W0619 00:11:48.932374 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 because the tablet was not found. No further retry: Not found: Tablet not found: 89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:48.932291 25725 raft_consensus.cc:325] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } }
I0619 00:11:48.932904 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 282 ms (attempt = 5)
I0619 00:11:48.933102 25725 raft_consensus.cc:351] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:48.933171 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.933256 25725 raft_consensus.cc:628] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:48.933739 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 because the tablet was not found. No further retry: Not found: Tablet not found: 89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:48.935251 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.936424 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.936420 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_a7e58e3f197748a2999b58edaa8e4fc6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:48.936976 25068 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 512 ms (attempt = 6)
W0619 00:11:48.937300 25068 catalog_manager.cc:2910] Async tablet task 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7507676a768e421dac27fe524538e0bb consensus state has no leader
W0619 00:11:48.937304 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a7e58e3f197748a2999b58edaa8e4fc6 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.938093 25724 tablet_bootstrap.cc:578] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:48.941875 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.941543 25725 consensus_queue.cc:227] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } }
W0619 00:11:48.942422 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 89a7ff9cdbc046779799e4e8f71adb42 already in progress: creating tablet
I0619 00:11:48.942656 25071 catalog_manager.cc:2891] Scheduling retry of 89a7ff9cdbc046779799e4e8f71adb42 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 44 ms (attempt = 2)
I0619 00:11:48.919378 25331 tablet_service.cc:759] Processing CreateTablet for tablet dc04126054714196af6bdf7e771ffab0 (table=default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.944036 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.920048 25330 tablet_service.cc:759] Processing CreateTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 (table=default.table_4a989339185d4e3fa0c743cabbcbf619 [id=f25f5cc3e1d64c95a0dcc4e36416c416]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.930470 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:48.923851 25326 tablet_service.cc:759] Processing CreateTablet for tablet ac129ed44f68432f800420175aa0a630 (table=default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.941459 25328 tablet_service.cc:759] Processing CreateTablet for tablet fb8e94875a8347a0a473c95a77c38795 (table=default.table_77bc6b25b36d4879ba67ea600dcc9114 [id=41f48f0e1f87464eb43c5bf2a36f70ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.925189 25327 tablet_service.cc:759] Processing CreateTablet for tablet a226a7e97a7e44af8301ed4e6363effb (table=default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:48.930424 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.945899 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 89a7ff9cdbc046779799e4e8f71adb42. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.944730 25325 tablet_service.cc:759] Processing CreateTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table=default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.948686 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dc04126054714196af6bdf7e771ffab0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.949064 25068 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 100 ms (attempt = 3)
W0619 00:11:48.949367 25068 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:48.952352 25698 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:48.952924 25725 ts_tablet_manager.cc:1073] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.069s	user 0.019s	sys 0.002s
I0619 00:11:48.953006 25698 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: This master's current role is: LEADER
I0619 00:11:48.953240 25725 ts_tablet_manager.cc:1028] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:48.953574 25725 tablet_bootstrap.cc:427] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:48.955772 25086 catalog_manager.cc:3663] Got report from deleted tablet 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) (replaced by tablet baf80fd968c44e679987eb4ebe4473de): Sending delete request for this tablet
I0619 00:11:48.956426 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
W0619 00:11:48.957888 25068 catalog_manager.cc:2910] Async tablet task 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 89a7ff9cdbc046779799e4e8f71adb42 consensus state has no leader
I0619 00:11:48.958914 25327 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a226a7e97a7e44af8301ed4e6363effb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.959779 25328 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fb8e94875a8347a0a473c95a77c38795. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.963744 25323 tablet_service.cc:759] Processing CreateTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table=default.table_a7e58e3f197748a2999b58edaa8e4fc6 [id=fa287205f19a461fa0ba005f9991513b]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:48.968240 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:48.971138 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:48.971565 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
W0619 00:11:48.972017 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6d6363f6403b4104b15b329ab7f0e483 already in progress: creating tablet
I0619 00:11:48.972285 25071 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 93 ms (attempt = 3)
I0619 00:11:48.972486 25332 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e3708e05f6054306b5bf7f1df2140e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.973675 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.974218 25736 tablet_replica.cc:261] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:48.974337 25467 tablet_service.cc:759] Processing CreateTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table=default.table_a7e58e3f197748a2999b58edaa8e4fc6 [id=fa287205f19a461fa0ba005f9991513b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:48.974598 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.974315 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_041edc89d445491abcd8368e840c196f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:48.974838 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.975018 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.974969 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 72e28bc6b57442b782d8aa07e1152cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.975214 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(7507676a768e421dac27fe524538e0bb)
W0619 00:11:48.975412 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_041edc89d445491abcd8368e840c196f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.975524 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.975767 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.975930 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:48.976142 25736 raft_consensus.cc:2042] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:48.976454 25736 raft_consensus.cc:2069] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:48.977389 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:48.977411 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:48.978765 25068 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 269 ms (attempt = 5)
I0619 00:11:48.978448 25603 tablet_service.cc:759] Processing CreateTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table=default.table_a7e58e3f197748a2999b58edaa8e4fc6 [id=fa287205f19a461fa0ba005f9991513b]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:48.979058 25068 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:48.978821 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_581afad997184fb0830cb18cbe8804c3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:48.979451 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 72e28bc6b57442b782d8aa07e1152cf4. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:48.979688 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_581afad997184fb0830cb18cbe8804c3 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.981717 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_48993904805645d49d4f0185d407c89a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:48.982537 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_48993904805645d49d4f0185d407c89a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:48.984129 25736 ts_tablet_manager.cc:1369] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:48.988790 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.989240 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:48.990568 25463 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.991080 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 128 ms (attempt = 4)
W0619 00:11:48.991413 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:48.992791 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:48.993031 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ac129ed44f68432f800420175aa0a630. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:48.994407 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC with a delay of 264 ms (attempt = 5)
W0619 00:11:48.994694 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:49.000461 25725 tablet_bootstrap.cc:578] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.000591 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc22ad98ed2e4e3eb2c62360626a6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.008796 25729 raft_consensus.cc:457] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.009068 25729 raft_consensus.cc:2732] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.595s (starting election)
I0619 00:11:49.009377 25336 tablet_service.cc:759] Processing CreateTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 (table=default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.009325 25729 raft_consensus.cc:479] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.010385 25729 leader_election.cc:243] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.010820 25729 leader_election.cc:243] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:49.013258 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a226a7e97a7e44af8301ed4e6363effb" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:49.014163 25604 tablet_service.cc:759] Processing CreateTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 (table=default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.014780 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60dfd91c98d94bfbbc6416268c65c378. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.014899 25558 leader_election.cc:306] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.015738 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 72e28bc6b57442b782d8aa07e1152cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.016136 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60dfd91c98d94bfbbc6416268c65c378. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.016573 25553 leader_election.cc:306] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Not found: Tablet not found: a226a7e97a7e44af8301ed4e6363effb
I0619 00:11:49.016877 25553 leader_election.cc:270] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.017607 25729 raft_consensus.cc:2732] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.572s (election complete)
I0619 00:11:49.017850 25729 raft_consensus.cc:2487] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.018738 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_a7e58e3f197748a2999b58edaa8e4fc6" } new_table_name: "default.table_95a48b4d8a824265bb7291a68eb33dc3" alter_external_catalogs: true
I0619 00:11:49.019560 25462 tablet_service.cc:759] Processing CreateTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 (table=default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.020162 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60dfd91c98d94bfbbc6416268c65c378. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.026473 25604 ts_tablet_manager.cc:1173] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.029382 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_5e925f129a1043b7ab80c112a0ee33e0" }
I0619 00:11:49.031249 25603 ts_tablet_manager.cc:1173] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.033143 25467 ts_tablet_manager.cc:1173] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.034515 25327 ts_tablet_manager.cc:1173] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.039654 25725 tablet_bootstrap.cc:427] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.040016 25725 ts_tablet_manager.cc:1045] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.087s	user 0.005s	sys 0.002s
I0619 00:11:49.041601 25725 raft_consensus.cc:325] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.042300 25725 raft_consensus.cc:351] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.042517 25725 raft_consensus.cc:628] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.047343 25336 ts_tablet_manager.cc:1173] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.047554 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:46325
--webserver_interface=127.24.119.61
--webserver_port=33317
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058 with env {}
I0619 00:11:49.048316 25332 ts_tablet_manager.cc:1173] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.049855 25463 tablet_service.cc:759] Processing CreateTablet for tablet 4c0614e9ed96420886243a6868f2e75e (table=default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.050467 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4c0614e9ed96420886243a6868f2e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.051384 25087 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 63 ms (attempt = 1)
W0619 00:11:49.051683 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.052608 25068 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 153 ms (attempt = 4)
W0619 00:11:49.052932 25068 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.055398 25604 tablet_service.cc:759] Processing CreateTablet for tablet 4c0614e9ed96420886243a6868f2e75e (table=default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.056715 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4c0614e9ed96420886243a6868f2e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.057323 25724 tablet_bootstrap.cc:427] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.057660 25724 ts_tablet_manager.cc:1045] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.132s	user 0.003s	sys 0.008s
I0619 00:11:49.059104 25724 raft_consensus.cc:325] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.059748 25724 raft_consensus.cc:351] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.059957 25724 raft_consensus.cc:628] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:49.043164 25725 consensus_queue.cc:227] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.060552 25724 consensus_queue.cc:227] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:49.063866 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.065773 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.066406 25729 raft_consensus.cc:457] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.066660 25729 raft_consensus.cc:2732] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.686s (starting election)
I0619 00:11:49.066937 25729 raft_consensus.cc:479] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.068373 25729 leader_election.cc:243] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:49.068764 25729 leader_election.cc:243] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.069109 25328 ts_tablet_manager.cc:1173] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.069186 25087 catalog_manager.cc:3663] Got report from deleted tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.069902 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.070010 25727 tablet_bootstrap.cc:578] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.071843 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ac129ed44f68432f800420175aa0a630" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:49.074420 25553 leader_election.cc:306] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Not found: Tablet not found: ac129ed44f68432f800420175aa0a630
I0619 00:11:49.074815 25603 tablet_service.cc:759] Processing CreateTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e (table=default.table_48993904805645d49d4f0185d407c89a [id=b46718e25afb4950af6cacec1d51a9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.073261 25336 tablet_service.cc:759] Processing CreateTablet for tablet 4c0614e9ed96420886243a6868f2e75e (table=default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:49.075273 25558 leader_election.cc:306] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.075522 25558 leader_election.cc:270] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.076690 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.076220 25729 raft_consensus.cc:2732] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.654s (election complete)
I0619 00:11:49.077628 25729 raft_consensus.cc:2487] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.078550 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.078982 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:49.079568 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7507676a768e421dac27fe524538e0bb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7507676a768e421dac27fe524538e0bb already in progress: creating tablet
I0619 00:11:49.079825 25067 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 48 ms (attempt = 1)
W0619 00:11:49.080204 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6d6363f6403b4104b15b329ab7f0e483 already in progress: creating tablet
I0619 00:11:49.080397 25067 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 60 ms (attempt = 1)
W0619 00:11:49.080922 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 because the tablet was not found. No further retry: Not found: Tablet not found: 985e23a32fa84104a06dd19d97117ab2
I0619 00:11:49.081388 25729 raft_consensus.cc:457] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.081601 25729 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.932s (starting election)
W0619 00:11:49.081737 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.081863 25729 raft_consensus.cc:479] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.082871 25729 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.084333 25729 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:49.084452 25467 tablet_service.cc:759] Processing CreateTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e (table=default.table_48993904805645d49d4f0185d407c89a [id=b46718e25afb4950af6cacec1d51a9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.085227 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1d10fb4561924e2e8d53c7a0b22a426e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.086344 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:49.087399 25086 catalog_manager.cc:3663] Got report from deleted tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.087816 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4c0614e9ed96420886243a6868f2e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.087946 25332 tablet_service.cc:759] Processing CreateTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e (table=default.table_48993904805645d49d4f0185d407c89a [id=b46718e25afb4950af6cacec1d51a9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.088106 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.088486 25332 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1d10fb4561924e2e8d53c7a0b22a426e. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.088454 25558 leader_election.cc:306] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.090903 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.076365 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1d10fb4561924e2e8d53c7a0b22a426e. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.092519 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.093081 25724 ts_tablet_manager.cc:1073] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.035s	user 0.003s	sys 0.001s
I0619 00:11:49.093478 25724 ts_tablet_manager.cc:1028] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.093868 25724 tablet_bootstrap.cc:427] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.096000 25724 tablet_bootstrap.cc:578] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:49.104008 25725 ts_tablet_manager.cc:1073] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.064s	user 0.003s	sys 0.002s
I0619 00:11:49.104624 25463 ts_tablet_manager.cc:1173] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.104979 25462 ts_tablet_manager.cc:1173] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.108183 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet dc04126054714196af6bdf7e771ffab0 because the tablet was not found. No further retry: Not found: Tablet not found: dc04126054714196af6bdf7e771ffab0
I0619 00:11:49.111814 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet baf80fd968c44e679987eb4ebe4473de
I0619 00:11:49.112146 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.112566 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.112964 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:49.117012 25543 tablet.cc:1678] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:11:49.119678 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 72 ms (attempt = 2)
W0619 00:11:49.120019 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.120604 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 276 ms (attempt = 5)
W0619 00:11:49.120914 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.121366 25335 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.123004 25462 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.104977 25725 ts_tablet_manager.cc:1028] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.124544 25725 tablet_bootstrap.cc:427] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:49.124989 25736 ts_tablet_manager.cc:1382] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.125246 25736 log.cc:986] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/7507676a768e421dac27fe524538e0bb
I0619 00:11:49.128275 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.132494 25335 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.116158 25602 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.133508 25604 ts_tablet_manager.cc:1173] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.133494 25334 ts_tablet_manager.cc:1173] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.134153 25730 tablet_replica.cc:261] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:49.134287 25727 log.cc:525] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Log is configured to *not* fsync() on all Append() calls
I0619 00:11:49.134383 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.134570 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.134778 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.134968 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.135176 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.135381 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.135562 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.135565 25332 ts_tablet_manager.cc:1173] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.135756 25730 raft_consensus.cc:2042] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.135994 25730 raft_consensus.cc:2069] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.137178 25730 ts_tablet_manager.cc:1369] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.138054 25603 ts_tablet_manager.cc:1173] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.138551 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 because the tablet was not found. No further retry: Not found: Tablet not found: 89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:49.135856 25331 ts_tablet_manager.cc:1173] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.141178 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.142871 25333 ts_tablet_manager.cc:1173] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.143841 25323 ts_tablet_manager.cc:1173] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.144217 25736 ts_tablet_manager.cc:1403] T 7507676a768e421dac27fe524538e0bb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:49.144524 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.144913 25725 tablet_bootstrap.cc:578] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:49.145180 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet baf80fd968c44e679987eb4ebe4473de with error code TABLET_NOT_RUNNING: Already present: State transition of tablet baf80fd968c44e679987eb4ebe4473de already in progress: creating tablet
I0619 00:11:49.145398 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 42 ms (attempt = 1)
I0619 00:11:49.148010 25467 ts_tablet_manager.cc:1173] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.145023 25330 ts_tablet_manager.cc:1173] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.153209 25730 ts_tablet_manager.cc:1382] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.153446 25730 log.cc:986] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/baf80fd968c44e679987eb4ebe4473de
W0619 00:11:49.158699 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6d6363f6403b4104b15b329ab7f0e483 already in progress: creating tablet
I0619 00:11:49.159011 25067 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 50 ms (attempt = 2)
I0619 00:11:49.159742 25725 tablet_bootstrap.cc:427] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.160032 25724 tablet_bootstrap.cc:427] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.160082 25725 ts_tablet_manager.cc:1045] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.036s	user 0.005s	sys 0.003s
I0619 00:11:49.160428 25724 ts_tablet_manager.cc:1045] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.067s	user 0.003s	sys 0.002s
W0619 00:11:49.160746 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet dc04126054714196af6bdf7e771ffab0 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet dc04126054714196af6bdf7e771ffab0 already in progress: creating tablet
I0619 00:11:49.161011 25071 catalog_manager.cc:2891] Scheduling retry of dc04126054714196af6bdf7e771ffab0 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 79 ms (attempt = 3)
W0619 00:11:49.161504 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 89a7ff9cdbc046779799e4e8f71adb42 already in progress: creating tablet
I0619 00:11:49.161743 25071 catalog_manager.cc:2891] Scheduling retry of 89a7ff9cdbc046779799e4e8f71adb42 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 108 ms (attempt = 3)
I0619 00:11:49.161898 25725 raft_consensus.cc:325] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.162318 25736 tablet_replica.cc:261] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:49.162466 25725 raft_consensus.cc:351] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.162694 25725 raft_consensus.cc:628] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.162591 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.163159 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.162874 25724 raft_consensus.cc:325] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.163475 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.163656 25724 raft_consensus.cc:351] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.163115 25725 consensus_queue.cc:227] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.163877 25724 raft_consensus.cc:628] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:49.165987 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.167438 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.168355 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.160002 25326 ts_tablet_manager.cc:1173] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.170267 25325 ts_tablet_manager.cc:1173] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.088915 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:49.171099 25336 ts_tablet_manager.cc:1173] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.171835 25730 ts_tablet_manager.cc:1403] T baf80fd968c44e679987eb4ebe4473de P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:49.169430 25724 consensus_queue.cc:227] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:49.173027 25553 leader_election.cc:306] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.173332 25553 leader_election.cc:270] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.169848 25725 ts_tablet_manager.cc:1073] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.001s
I0619 00:11:49.175324 25725 ts_tablet_manager.cc:1028] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.175724 25725 tablet_bootstrap.cc:427] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:49.176185 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.175271 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.177073 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.177093 25729 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.622s (election complete)
I0619 00:11:49.177367 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.177320 25729 raft_consensus.cc:2487] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.177585 25736 raft_consensus.cc:2042] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.177896 25736 raft_consensus.cc:2069] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:49.181107 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.182678 25727 tablet_bootstrap.cc:427] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:49.182878 25736 ts_tablet_manager.cc:1369] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.183039 25727 ts_tablet_manager.cc:1045] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.279s	user 0.005s	sys 0.016s
I0619 00:11:49.185009 25725 tablet_bootstrap.cc:578] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.185745 25724 ts_tablet_manager.cc:1073] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.000s
I0619 00:11:49.186944 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) successfully deleted
I0619 00:11:49.187113 25724 ts_tablet_manager.cc:1028] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.187451 25724 tablet_bootstrap.cc:427] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.188335 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.189319 25336 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:49.190277 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet baf80fd968c44e679987eb4ebe4473de with error code TABLET_NOT_RUNNING: Already present: State transition of tablet baf80fd968c44e679987eb4ebe4473de already in progress: creating tablet
I0619 00:11:49.190501 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 81 ms (attempt = 2)
I0619 00:11:49.190948 25724 tablet_bootstrap.cc:578] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:49.193112 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 98 ms (attempt = 3)
W0619 00:11:49.193357 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.195454 25725 tablet_bootstrap.cc:427] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.195796 25725 ts_tablet_manager.cc:1045] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.020s	user 0.005s	sys 0.000s
I0619 00:11:49.196214 25736 ts_tablet_manager.cc:1382] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.196442 25736 log.cc:986] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:49.196893 25736 ts_tablet_manager.cc:1403] T 6d6363f6403b4104b15b329ab7f0e483 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:49.197105 25725 raft_consensus.cc:325] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.197741 25725 raft_consensus.cc:351] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.197921 25725 raft_consensus.cc:628] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.198519 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) successfully deleted
I0619 00:11:49.198365 25725 consensus_queue.cc:227] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:49.201428 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.201629 25087 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 1 got 0
I0619 00:11:49.202284 25725 ts_tablet_manager.cc:1073] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
I0619 00:11:49.202365 25087 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 62 ms (attempt = 1)
W0619 00:11:49.202646 25087 catalog_manager.cc:3945] Failed to send e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
W0619 00:11:49.204102 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.205766 25724 tablet_bootstrap.cc:427] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.206080 25724 ts_tablet_manager.cc:1045] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.005s
I0619 00:11:49.206835 25068 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 259 ms (attempt = 5)
W0619 00:11:49.207105 25068 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.207530 25725 ts_tablet_manager.cc:1028] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.207962 25725 tablet_bootstrap.cc:427] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:49.208104 25724 raft_consensus.cc:325] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.208727 25724 raft_consensus.cc:351] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.208933 25724 raft_consensus.cc:628] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:49.209432 25724 consensus_queue.cc:227] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.209983 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.210198 25725 tablet_bootstrap.cc:578] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.211629 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:49.212790 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6d6363f6403b4104b15b329ab7f0e483 already in progress: creating tablet
I0619 00:11:49.213023 25067 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 76 ms (attempt = 3)
W0619 00:11:49.213090 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.213713 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 because the tablet was not found. No further retry: Not found: Tablet not found: 6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:49.214273 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 1 got 0
W0619 00:11:49.214751 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 because the tablet was not found. No further retry: Not found: Tablet not found: 6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:49.215039 25087 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 19 ms (attempt = 1)
W0619 00:11:49.215381 25087 catalog_manager.cc:3945] Failed to send fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
W0619 00:11:49.215821 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet baf80fd968c44e679987eb4ebe4473de with error code TABLET_NOT_RUNNING: Already present: State transition of tablet baf80fd968c44e679987eb4ebe4473de already in progress: creating tablet
I0619 00:11:49.216058 25071 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
W0619 00:11:49.216161 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
W0619 00:11:49.217375 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.219380 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 1 got 0
I0619 00:11:49.220126 25087 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 48 ms (attempt = 1)
W0619 00:11:49.220454 25087 catalog_manager.cc:3945] Failed to send fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.221905 25727 raft_consensus.cc:325] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } }
I0619 00:11:49.222751 25727 raft_consensus.cc:351] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.222951 25727 raft_consensus.cc:628] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.223289 25724 ts_tablet_manager.cc:1073] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.000s
I0619 00:11:49.223678 25724 ts_tablet_manager.cc:1028] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.224030 25724 tablet_bootstrap.cc:427] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.226235 25724 tablet_bootstrap.cc:578] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:49.135355 25683 outbound_call.cc:287] RPC callback for RPC call kudu.master.MasterService.AlterTable -> {remote=127.24.119.62:37587, user_credentials={real_user=jenkins}} blocked reactor thread for 45124.8us
I0619 00:11:49.232105 25725 tablet_bootstrap.cc:427] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.232430 25725 ts_tablet_manager.cc:1045] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.025s	user 0.002s	sys 0.007s
I0619 00:11:49.232395 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_9bbf1c0d460e4aafa3551e40c25513c4" }
I0619 00:11:49.233353 25085 master_service.cc:523] ReplaceTablet: received request to replace tablet 7507676a768e421dac27fe524538e0bb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:49.234447 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_15724e9cddcb457a8edaebf78d6afc15"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:49.235483 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_15724e9cddcb457a8edaebf78d6afc15 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.235677 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 39 ms (attempt = 2)
W0619 00:11:49.235965 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.234436 25086 master_service.cc:523] ReplaceTablet: received request to replace tablet a226a7e97a7e44af8301ed4e6363effb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:49.234705 25727 consensus_queue.cc:227] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } }
I0619 00:11:49.232522 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c297d9ff075944df87aac8f56d2f9d95" } new_table_name: "default.table_84d38522d00a488c82a136b79665c4cd" alter_external_catalogs: true
I0619 00:11:49.233814 25725 raft_consensus.cc:325] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.238091 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_dda28a7d27fb424e8e0e9f6657a52c6c"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:49.238626 25725 raft_consensus.cc:351] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.238926 25725 raft_consensus.cc:628] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:49.239112 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dda28a7d27fb424e8e0e9f6657a52c6c in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.239497 25725 consensus_queue.cc:227] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.237094 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_aecb1278dc98420aa83aaae47fb56937"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:49.242980 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aecb1278dc98420aa83aaae47fb56937 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.243331 25085 catalog_manager.cc:3663] Got report from deleted tablet dc04126054714196af6bdf7e771ffab0 (table default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.244011 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.244750 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.244952 25729 raft_consensus.cc:457] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:49.245098 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.245214 25729 raft_consensus.cc:2732] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.512s (starting election)
I0619 00:11:49.245489 25729 raft_consensus.cc:479] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.245982 25467 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.246405 25463 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.246480 25729 leader_election.cc:243] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.246888 25729 leader_election.cc:243] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:49.247098 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet dc04126054714196af6bdf7e771ffab0 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet dc04126054714196af6bdf7e771ffab0 already in progress: creating tablet
I0619 00:11:49.247321 25071 catalog_manager.cc:2891] Scheduling retry of dc04126054714196af6bdf7e771ffab0 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 26 ms (attempt = 1)
W0619 00:11:49.248018 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet dc04126054714196af6bdf7e771ffab0 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet dc04126054714196af6bdf7e771ffab0 already in progress: creating tablet
I0619 00:11:49.248265 25071 catalog_manager.cc:2891] Scheduling retry of dc04126054714196af6bdf7e771ffab0 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 162 ms (attempt = 4)
I0619 00:11:49.248327 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc22ad98ed2e4e3eb2c62360626a6f87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:49.235455 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b4370a7e80854fd8965e81d09a545d89"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:49.248853 25068 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
W0619 00:11:49.249332 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b4370a7e80854fd8965e81d09a545d89 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.249678 25085 catalog_manager.cc:3663] Got report from deleted tablet 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) (replaced by tablet baf80fd968c44e679987eb4ebe4473de): Sending delete request for this tablet
I0619 00:11:49.250360 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.251147 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc22ad98ed2e4e3eb2c62360626a6f87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:49.251674 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.252717 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:49.252876 25558 leader_election.cc:306] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.254434 25085 catalog_manager.cc:3663] Got report from deleted tablet dc04126054714196af6bdf7e771ffab0 (table default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.255072 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
W0619 00:11:49.255069 25553 leader_election.cc:306] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.255318 25553 leader_election.cc:270] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.255439 25725 ts_tablet_manager.cc:1073] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.000s
W0619 00:11:49.255837 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7507676a768e421dac27fe524538e0bb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7507676a768e421dac27fe524538e0bb already in progress: creating tablet
I0619 00:11:49.256006 25729 raft_consensus.cc:2732] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.185s (election complete)
I0619 00:11:49.256044 25067 catalog_manager.cc:2891] Scheduling retry of 7507676a768e421dac27fe524538e0bb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 35 ms (attempt = 1)
I0619 00:11:49.256219 25729 raft_consensus.cc:2487] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.256456 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet fc22ad98ed2e4e3eb2c62360626a6f87
I0619 00:11:49.256768 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.257614 25463 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.257139 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.258188 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.258178 25336 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.258764 25463 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.259097 25736 tablet_replica.cc:261] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:49.259346 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(dc04126054714196af6bdf7e771ffab0)
W0619 00:11:49.259323 25067 catalog_manager.cc:2910] Async tablet task baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet baf80fd968c44e679987eb4ebe4473de consensus state has no leader
I0619 00:11:49.259567 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(dc04126054714196af6bdf7e771ffab0)
W0619 00:11:49.259654 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:49.259786 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:49.259856 25067 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 62 ms (attempt = 1)
I0619 00:11:49.259969 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:49.260208 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:49.260270 25604 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.260402 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:49.260586 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:49.260779 25730 tablet_replica.cc:261] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:49.260782 25736 raft_consensus.cc:2042] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.261013 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.261049 25736 raft_consensus.cc:2069] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.261210 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.261422 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.261618 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(fc22ad98ed2e4e3eb2c62360626a6f87)
W0619 00:11:49.261760 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.261865 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.262050 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.262255 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.262421 25730 raft_consensus.cc:2042] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.262698 25730 raft_consensus.cc:2069] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.255807 25725 ts_tablet_manager.cc:1028] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.263412 25736 ts_tablet_manager.cc:1369] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.264523 25730 ts_tablet_manager.cc:1369] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.264953 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet a226a7e97a7e44af8301ed4e6363effb
I0619 00:11:49.265244 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a226a7e97a7e44af8301ed4e6363effb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:49.265643 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a226a7e97a7e44af8301ed4e6363effb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:49.266041 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a226a7e97a7e44af8301ed4e6363effb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:49.266356 25086 catalog_manager.cc:4596] ReplaceTablet: tablet a226a7e97a7e44af8301ed4e6363effb deleted and replaced by tablet 36f5eec972454080b7b434b4df7e62e3
I0619 00:11:49.266367 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 47 ms (attempt = 2)
W0619 00:11:49.267659 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.267902 25604 tablet_service.cc:800] Processing DeleteTablet for tablet a226a7e97a7e44af8301ed4e6363effb with delete_type TABLET_DATA_DELETED (replaced by tablet 36f5eec972454080b7b434b4df7e62e3) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.268379 25336 tablet_service.cc:800] Processing DeleteTablet for tablet a226a7e97a7e44af8301ed4e6363effb with delete_type TABLET_DATA_DELETED (replaced by tablet 36f5eec972454080b7b434b4df7e62e3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.268931 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 66 ms (attempt = 2)
W0619 00:11:49.269209 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.269176 25463 tablet_service.cc:800] Processing DeleteTablet for tablet a226a7e97a7e44af8301ed4e6363effb with delete_type TABLET_DATA_DELETED (replaced by tablet 36f5eec972454080b7b434b4df7e62e3) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:49.270368 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a226a7e97a7e44af8301ed4e6363effb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a226a7e97a7e44af8301ed4e6363effb already in progress: creating tablet
I0619 00:11:49.270654 25067 catalog_manager.cc:2891] Scheduling retry of a226a7e97a7e44af8301ed4e6363effb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 62 ms (attempt = 1)
I0619 00:11:49.270961 25727 ts_tablet_manager.cc:1073] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.084s	user 0.012s	sys 0.002s
I0619 00:11:49.271335 25727 ts_tablet_manager.cc:1028] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:49.271852 25727 tablet_bootstrap.cc:427] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:49.273237 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.273421 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.274036 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.274226 25336 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.275029 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:49.275410 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet baf80fd968c44e679987eb4ebe4473de with error code TABLET_NOT_RUNNING: Already present: State transition of tablet baf80fd968c44e679987eb4ebe4473de already in progress: creating tablet
I0619 00:11:49.275740 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 88 ms (attempt = 3)
I0619 00:11:49.275861 25463 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.276255 25725 tablet_bootstrap.cc:427] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:49.276312 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 96 ms (attempt = 3)
W0619 00:11:49.276609 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.278230 25727 tablet_bootstrap.cc:578] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.279067 25725 tablet_bootstrap.cc:578] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.280994 25080 catalog_manager.cc:3663] Got report from deleted tablet 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) (replaced by tablet baf80fd968c44e679987eb4ebe4473de): Sending delete request for this tablet
I0619 00:11:49.281740 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.285908 25724 tablet_bootstrap.cc:427] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.286278 25724 ts_tablet_manager.cc:1045] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.062s	user 0.004s	sys 0.005s
I0619 00:11:49.281340 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.287746 25724 raft_consensus.cc:325] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.288573 25724 raft_consensus.cc:351] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.288817 25724 raft_consensus.cc:628] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:49.282050 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.282781 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.289299 25724 consensus_queue.cc:227] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.289846 25741 tablet_replica.cc:261] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:49.289922 25463 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.290117 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:49.290343 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:49.290514 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:49.290712 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:49.290916 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:49.291088 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:49.291254 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(7507676a768e421dac27fe524538e0bb)
I0619 00:11:49.291447 25741 raft_consensus.cc:2042] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.291762 25741 raft_consensus.cc:2069] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.292276 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.292706 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7507676a768e421dac27fe524538e0bb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.293154 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 143 ms (attempt = 4)
W0619 00:11:49.293385 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.291383 25730 ts_tablet_manager.cc:1382] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.293781 25730 log.cc:986] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/fc22ad98ed2e4e3eb2c62360626a6f87
I0619 00:11:49.294340 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.295058 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 7507676a768e421dac27fe524538e0bb with delete_type TABLET_DATA_DELETED (replaced by tablet baf80fd968c44e679987eb4ebe4473de) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.295246 25730 ts_tablet_manager.cc:1403] T fc22ad98ed2e4e3eb2c62360626a6f87 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:11:49.297289 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.299098 25741 ts_tablet_manager.cc:1369] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.299064 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]) successfully deleted
I0619 00:11:49.299901 25736 ts_tablet_manager.cc:1382] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.300586 25730 tablet_replica.cc:261] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:49.300889 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.301105 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.301316 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.300231 25725 tablet_bootstrap.cc:427] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.302525 25604 tablet_service.cc:759] Processing CreateTablet for tablet 36f5eec972454080b7b434b4df7e62e3 (table=default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.302916 25725 ts_tablet_manager.cc:1045] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.039s	user 0.005s	sys 0.001s
I0619 00:11:49.303125 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 36f5eec972454080b7b434b4df7e62e3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.304338 25725 raft_consensus.cc:325] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.305040 25725 raft_consensus.cc:351] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.305260 25725 raft_consensus.cc:628] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.305465 25336 tablet_service.cc:759] Processing CreateTablet for tablet 36f5eec972454080b7b434b4df7e62e3 (table=default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.306051 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 36f5eec972454080b7b434b4df7e62e3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.306052 25463 tablet_service.cc:759] Processing CreateTablet for tablet 36f5eec972454080b7b434b4df7e62e3 (table=default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.306627 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 36f5eec972454080b7b434b4df7e62e3. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.308498 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.310281 25725 consensus_queue.cc:227] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.311213 25724 ts_tablet_manager.cc:1073] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.000s
I0619 00:11:49.313684 25724 ts_tablet_manager.cc:1028] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.314129 25080 catalog_manager.cc:3663] Got report from deleted tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.314807 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.315253 25336 ts_tablet_manager.cc:1173] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.315482 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 65 ms (attempt = 3)
W0619 00:11:49.315817 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
W0619 00:11:49.316294 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.317131 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.317342 25080 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_48993904805645d49d4f0185d407c89a" } new_table_name: "default.table_f4fd4524d920428fa515c1266d90fb2a" alter_external_catalogs: true
I0619 00:11:49.318403 25724 tablet_bootstrap.cc:427] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.322566 25463 ts_tablet_manager.cc:1173] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.323715 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.323694 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.325379 25086 catalog_manager.cc:3663] Got report from deleted tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.326063 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.326810 25325 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.302094 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.327571 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.327832 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.327972 25736 log.cc:986] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/dc04126054714196af6bdf7e771ffab0
I0619 00:11:49.328047 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.328223 25730 raft_consensus.cc:2042] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.328492 25730 raft_consensus.cc:2069] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.328899 25725 ts_tablet_manager.cc:1073] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.000s
I0619 00:11:49.329295 25725 ts_tablet_manager.cc:1028] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.331974 25730 ts_tablet_manager.cc:1369] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.332113 25724 tablet_bootstrap.cc:578] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:49.333268 25741 ts_tablet_manager.cc:1382] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.333539 25741 log.cc:986] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/7507676a768e421dac27fe524538e0bb
I0619 00:11:49.333199 25736 ts_tablet_manager.cc:1403] T dc04126054714196af6bdf7e771ffab0 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:49.334045 25741 ts_tablet_manager.cc:1403] T 7507676a768e421dac27fe524538e0bb P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:49.335705 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet dc04126054714196af6bdf7e771ffab0 (table default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]) successfully deleted
I0619 00:11:49.335878 25604 ts_tablet_manager.cc:1173] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.336514 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a226a7e97a7e44af8301ed4e6363effb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:49.336818 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.336918 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 80 ms (attempt = 3)
W0619 00:11:49.337206 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_c297d9ff075944df87aac8f56d2f9d95 [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.338634 25727 tablet_bootstrap.cc:427] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:49.338944 25727 ts_tablet_manager.cc:1045] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.067s	user 0.004s	sys 0.003s
I0619 00:11:49.339870 25087 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 27 ms (attempt = 1)
W0619 00:11:49.340149 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.340175 25727 raft_consensus.cc:325] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.340798 25727 raft_consensus.cc:351] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.340984 25727 raft_consensus.cc:628] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.342057 25727 consensus_queue.cc:227] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:49.342439 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:49.342684 25071 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 53 ms (attempt = 1)
I0619 00:11:49.337203 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) successfully deleted
W0619 00:11:49.345522 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6d6363f6403b4104b15b329ab7f0e483 already in progress: creating tablet
I0619 00:11:49.345758 25067 catalog_manager.cc:2891] Scheduling retry of 6d6363f6403b4104b15b329ab7f0e483 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 165 ms (attempt = 4)
W0619 00:11:49.346555 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.346547 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7507676a768e421dac27fe524538e0bb because the tablet was not found. No further retry: Not found: Tablet not found: 7507676a768e421dac27fe524538e0bb
I0619 00:11:49.346524 25336 tablet_service.cc:800] Processing DeleteTablet for tablet a226a7e97a7e44af8301ed4e6363effb with delete_type TABLET_DATA_DELETED (replaced by tablet 36f5eec972454080b7b434b4df7e62e3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.346915 25725 tablet_bootstrap.cc:427] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:49.348268 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.348966 25727 ts_tablet_manager.cc:1073] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.002s
I0619 00:11:49.349306 25727 ts_tablet_manager.cc:1028] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:49.349669 25727 tablet_bootstrap.cc:427] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:49.351902 25736 tablet_replica.cc:261] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:49.352164 25727 tablet_bootstrap.cc:578] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.353339 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.353564 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.355760 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.355988 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.356241 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.356343 25087 catalog_manager.cc:3663] Got report from deleted tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
W0619 00:11:49.356546 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:49.356802 25067 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 2)
I0619 00:11:49.357040 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
W0619 00:11:49.357187 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a226a7e97a7e44af8301ed4e6363effb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a226a7e97a7e44af8301ed4e6363effb already in progress: creating tablet
I0619 00:11:49.357481 25724 tablet_bootstrap.cc:427] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.357903 25724 ts_tablet_manager.cc:1045] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.040s	user 0.006s	sys 0.000s
I0619 00:11:49.359364 25724 raft_consensus.cc:325] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.360046 25724 raft_consensus.cc:351] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.360262 25724 raft_consensus.cc:628] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:49.360822 25724 consensus_queue.cc:227] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.362471 25080 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 47 ms (attempt = 1)
W0619 00:11:49.362783 25080 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.363709 25724 ts_tablet_manager.cc:1073] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I0619 00:11:49.364472 25724 ts_tablet_manager.cc:1028] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.364856 25724 tablet_bootstrap.cc:427] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:49.366808 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.368013 25082 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]). Expected version 1 got 0
I0619 00:11:49.368242 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 70 ms (attempt = 2)
W0619 00:11:49.368582 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.368793 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.368808 25082 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 45 ms (attempt = 1)
W0619 00:11:49.369072 25082 catalog_manager.cc:3945] Failed to send 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.371309 25604 tablet_service.cc:759] Processing CreateTablet for tablet aaf62da07a9740c1aa44359d3f604513 (table=default.table_dda28a7d27fb424e8e0e9f6657a52c6c [id=f181d18c1f844c83a0640004333ae473]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.371775 25727 tablet_bootstrap.cc:427] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:49.372087 25727 ts_tablet_manager.cc:1045] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.002s
I0619 00:11:49.372206 25463 tablet_service.cc:759] Processing CreateTablet for tablet aaf62da07a9740c1aa44359d3f604513 (table=default.table_dda28a7d27fb424e8e0e9f6657a52c6c [id=f181d18c1f844c83a0640004333ae473]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.372876 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aaf62da07a9740c1aa44359d3f604513. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.373291 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 169 ms (attempt = 4)
W0619 00:11:49.373622 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.373719 25603 tablet_service.cc:759] Processing CreateTablet for tablet 292b164659e343158de8e7e3eeb9be31 (table=default.table_aecb1278dc98420aa83aaae47fb56937 [id=e2a7900f428948bab0aefd132d8874d4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.374508 25467 tablet_service.cc:759] Processing CreateTablet for tablet 292b164659e343158de8e7e3eeb9be31 (table=default.table_aecb1278dc98420aa83aaae47fb56937 [id=e2a7900f428948bab0aefd132d8874d4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.357393 25067 catalog_manager.cc:2891] Scheduling retry of a226a7e97a7e44af8301ed4e6363effb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 34 ms (attempt = 2)
I0619 00:11:49.376886 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.375452 25727 raft_consensus.cc:325] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.377418 25727 raft_consensus.cc:351] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.357640 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.378187 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.378403 25736 raft_consensus.cc:2042] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.378489 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.378645 25736 raft_consensus.cc:2069] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.378908 25325 tablet_service.cc:759] Processing CreateTablet for tablet aaf62da07a9740c1aa44359d3f604513 (table=default.table_dda28a7d27fb424e8e0e9f6657a52c6c [id=f181d18c1f844c83a0640004333ae473]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.379201 25336 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.379503 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aaf62da07a9740c1aa44359d3f604513. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.379667 25741 tablet_replica.cc:261] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:49.379885 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.380070 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.380131 25736 ts_tablet_manager.cc:1369] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.380249 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.380131 25326 tablet_service.cc:759] Processing CreateTablet for tablet 292b164659e343158de8e7e3eeb9be31 (table=default.table_aecb1278dc98420aa83aaae47fb56937 [id=e2a7900f428948bab0aefd132d8874d4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.380767 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 292b164659e343158de8e7e3eeb9be31. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.381016 25725 tablet_bootstrap.cc:578] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.381722 25082 catalog_manager.cc:3663] Got report from deleted tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.382370 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.384259 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 292b164659e343158de8e7e3eeb9be31. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.384428 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 171 ms (attempt = 4)
W0619 00:11:49.384722 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.385669 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.385761 25727 raft_consensus.cc:628] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.386248 25727 consensus_queue.cc:227] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.386782 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.387847 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.388057 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.389379 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(6d6363f6403b4104b15b329ab7f0e483)
I0619 00:11:49.389600 25741 raft_consensus.cc:2042] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.389919 25741 raft_consensus.cc:2069] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:49.390974 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.395426 25462 tablet_service.cc:759] Processing CreateTablet for tablet 3b80a302aacb418091aee54ab4be7249 (table=default.table_b4370a7e80854fd8965e81d09a545d89 [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.395668 25469 tablet_service.cc:759] Processing CreateTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 (table=default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.396033 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3b80a302aacb418091aee54ab4be7249. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.397377 25336 tablet_service.cc:759] Processing CreateTablet for tablet 3b80a302aacb418091aee54ab4be7249 (table=default.table_b4370a7e80854fd8965e81d09a545d89 [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.397418 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 536 ms (attempt = 6)
W0619 00:11:49.397718 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.398308 25330 tablet_service.cc:759] Processing CreateTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 (table=default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.398922 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.399868 25470 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:49.400321 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.399879 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 53688f0e4a604bc4a07dec9278a8dad6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.400709 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.401919 25082 catalog_manager.cc:3663] Got report from deleted tablet a226a7e97a7e44af8301ed4e6363effb (table default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3): Sending delete request for this tablet
I0619 00:11:49.402554 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a226a7e97a7e44af8301ed4e6363effb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:49.403267 25323 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.403759 25467 ts_tablet_manager.cc:1173] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.387107 25724 tablet_bootstrap.cc:578] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:49.391109 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aaf62da07a9740c1aa44359d3f604513. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.404498 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a226a7e97a7e44af8301ed4e6363effb with delete_type TABLET_DATA_DELETED (replaced by tablet 36f5eec972454080b7b434b4df7e62e3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.398726 25601 tablet_service.cc:759] Processing CreateTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 (table=default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.405112 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 292b164659e343158de8e7e3eeb9be31. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.398703 25602 tablet_service.cc:759] Processing CreateTablet for tablet 3b80a302aacb418091aee54ab4be7249 (table=default.table_b4370a7e80854fd8965e81d09a545d89 [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.406059 25725 tablet_bootstrap.cc:427] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.406402 25725 ts_tablet_manager.cc:1045] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.077s	user 0.006s	sys 0.003s
I0619 00:11:49.406731 25082 catalog_manager.cc:3663] Got report from deleted tablet a226a7e97a7e44af8301ed4e6363effb (table default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3): Sending delete request for this tablet
I0619 00:11:49.407369 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a226a7e97a7e44af8301ed4e6363effb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:49.397449 25741 ts_tablet_manager.cc:1369] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.391165 25730 ts_tablet_manager.cc:1382] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.408243 25730 log.cc:986] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a226a7e97a7e44af8301ed4e6363effb
I0619 00:11:49.408769 25730 ts_tablet_manager.cc:1403] T a226a7e97a7e44af8301ed4e6363effb P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:49.408843 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a226a7e97a7e44af8301ed4e6363effb with delete_type TABLET_DATA_DELETED (replaced by tablet 36f5eec972454080b7b434b4df7e62e3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.399925 25727 ts_tablet_manager.cc:1073] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.000s
I0619 00:11:49.407979 25725 raft_consensus.cc:325] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.409704 25725 raft_consensus.cc:351] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.409847 25727 ts_tablet_manager.cc:1028] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:49.409920 25725 raft_consensus.cc:628] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.410753 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 53688f0e4a604bc4a07dec9278a8dad6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.410902 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a226a7e97a7e44af8301ed4e6363effb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:49.411765 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a226a7e97a7e44af8301ed4e6363effb with delete_type TABLET_DATA_DELETED (replaced by tablet 36f5eec972454080b7b434b4df7e62e3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.412334 25727 tablet_bootstrap.cc:427] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:49.413864 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3b80a302aacb418091aee54ab4be7249. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.413836 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.417269 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 53688f0e4a604bc4a07dec9278a8dad6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.417819 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3b80a302aacb418091aee54ab4be7249. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.421473 25601 ts_tablet_manager.cc:1173] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.410405 25725 consensus_queue.cc:227] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.422916 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 156 ms (attempt = 4)
W0619 00:11:49.423202 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
W0619 00:11:49.424973 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.426815 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 67 ms (attempt = 2)
W0619 00:11:49.427129 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.427462 25067 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 40 ms (attempt = 2)
W0619 00:11:49.427795 25067 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.429184 25082 catalog_manager.cc:3663] Got report from deleted tablet baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:49.429913 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.429214 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet a226a7e97a7e44af8301ed4e6363effb (table default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]) successfully deleted
I0619 00:11:49.432416 25470 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.433663 25462 ts_tablet_manager.cc:1173] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.434020 25467 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.434561 25463 ts_tablet_manager.cc:1173] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.435382 25469 ts_tablet_manager.cc:1173] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.436975 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 268 ms (attempt = 5)
W0619 00:11:49.437237 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.439182 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_041edc89d445491abcd8368e840c196f" }
I0619 00:11:49.442011 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 106 ms (attempt = 3)
W0619 00:11:49.442270 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.446128 25725 ts_tablet_manager.cc:1073] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.040s	user 0.003s	sys 0.001s
I0619 00:11:49.446504 25725 ts_tablet_manager.cc:1028] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.446893 25725 tablet_bootstrap.cc:427] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:49.447463 25604 ts_tablet_manager.cc:1173] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.448649 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.448948 25602 ts_tablet_manager.cc:1173] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.449939 25068 catalog_manager.cc:2910] Async tablet task 7507676a768e421dac27fe524538e0bb (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7507676a768e421dac27fe524538e0bb consensus state has no leader
I0619 00:11:49.449966 25603 ts_tablet_manager.cc:1173] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.451818 25725 tablet_bootstrap.cc:578] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.453927 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 60dfd91c98d94bfbbc6416268c65c378
I0619 00:11:49.454200 25736 ts_tablet_manager.cc:1382] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.454242 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 60dfd91c98d94bfbbc6416268c65c378 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.454386 25724 tablet_bootstrap.cc:427] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.454435 25736 log.cc:986] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a226a7e97a7e44af8301ed4e6363effb
I0619 00:11:49.454653 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 60dfd91c98d94bfbbc6416268c65c378 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.454912 25736 ts_tablet_manager.cc:1403] T a226a7e97a7e44af8301ed4e6363effb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:49.455034 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 60dfd91c98d94bfbbc6416268c65c378 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.454793 25724 ts_tablet_manager.cc:1045] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.090s	user 0.007s	sys 0.001s
I0619 00:11:49.456377 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.455546 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.457198 25604 tablet_service.cc:800] Processing DeleteTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.457695 25727 tablet_bootstrap.cc:578] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.458118 25730 tablet_replica.cc:261] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:49.458371 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.457999 25724 raft_consensus.cc:325] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.458566 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.458837 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.459041 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.459237 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.459403 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.426517 25330 ts_tablet_manager.cc:1173] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.459573 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.459733 25736 tablet_replica.cc:261] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:49.459766 25730 raft_consensus.cc:2042] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.459955 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:49.459915 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet a226a7e97a7e44af8301ed4e6363effb (table default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]) successfully deleted
I0619 00:11:49.460016 25730 raft_consensus.cc:2069] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.460140 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:49.460423 25325 ts_tablet_manager.cc:1173] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.460750 25326 ts_tablet_manager.cc:1173] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.460497 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:49.461001 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:49.461211 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:49.461395 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:49.461334 25336 ts_tablet_manager.cc:1173] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.461565 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:49.461758 25736 raft_consensus.cc:2042] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.461632 25730 ts_tablet_manager.cc:1369] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.462002 25736 raft_consensus.cc:2069] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.463199 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_581afad997184fb0830cb18cbe8804c3" }
I0619 00:11:49.467241 25068 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 532 ms (attempt = 6)
W0619 00:11:49.467533 25068 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.467556 25724 raft_consensus.cc:351] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.467833 25724 raft_consensus.cc:628] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:49.468729 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet ac129ed44f68432f800420175aa0a630 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:49.468907 25067 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 110 ms (attempt = 3)
I0619 00:11:49.469048 25736 ts_tablet_manager.cc:1369] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:49.469228 25067 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.472786 25730 ts_tablet_manager.cc:1382] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.473934 25741 ts_tablet_manager.cc:1382] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.474166 25741 log.cc:986] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:49.474731 25741 ts_tablet_manager.cc:1403] T 6d6363f6403b4104b15b329ab7f0e483 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:49.468401 25724 consensus_queue.cc:227] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.479184 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 3b80a302aacb418091aee54ab4be7249 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:49.485132 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.485570 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 4c0614e9ed96420886243a6868f2e75e
I0619 00:11:49.485795 25083 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]). Expected version 1 got 0
I0619 00:11:49.485916 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.486554 25083 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 29 ms (attempt = 1)
W0619 00:11:49.486877 25083 catalog_manager.cc:3945] Failed to send 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.486938 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:49.487126 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.490228 25724 ts_tablet_manager.cc:1073] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.034s	user 0.004s	sys 0.000s
I0619 00:11:49.490600 25724 ts_tablet_manager.cc:1028] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.491266 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet ac129ed44f68432f800420175aa0a630
I0619 00:11:49.490984 25724 tablet_bootstrap.cc:427] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.491551 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:49.491978 25081 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]). Expected version 1 got 0
I0619 00:11:49.492725 25081 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 27 ms (attempt = 1)
W0619 00:11:49.492987 25081 catalog_manager.cc:3945] Failed to send 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.493919 25469 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.492892 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_62055e93837844cd9a5dc2a343a23878"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:49.494554 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_62055e93837844cd9a5dc2a343a23878 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.486670 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.495723 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.496222 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 79 ms (attempt = 3)
W0619 00:11:49.496507 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.491989 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:49.497442 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:49.497790 25080 catalog_manager.cc:4596] ReplaceTablet: tablet ac129ed44f68432f800420175aa0a630 deleted and replaced by tablet 35acf55f35064c1491122a13ff9f5744
I0619 00:11:49.498270 25724 tablet_bootstrap.cc:578] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:49.500097 25730 log.cc:986] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/60dfd91c98d94bfbbc6416268c65c378
I0619 00:11:49.500502 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 3b80a302aacb418091aee54ab4be7249
I0619 00:11:49.500574 25730 ts_tablet_manager.cc:1403] T 60dfd91c98d94bfbbc6416268c65c378 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:49.500802 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:49.501155 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:49.501514 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:49.501832 25087 catalog_manager.cc:4596] ReplaceTablet: tablet 3b80a302aacb418091aee54ab4be7249 deleted and replaced by tablet acc89522b693498c987ca5d8ef8275ea
I0619 00:11:49.502166 25741 tablet_replica.cc:261] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:49.502214 25604 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.502393 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.502637 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.502866 25604 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.502849 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.503216 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.503321 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.503523 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.503742 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.503954 25730 tablet_replica.cc:261] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:49.503428 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d8b3984991a442fe95208294f57d466f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:49.504324 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(a226a7e97a7e44af8301ed4e6363effb)
I0619 00:11:49.504240 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:49.504498 25741 raft_consensus.cc:2042] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
W0619 00:11:49.504739 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d8b3984991a442fe95208294f57d466f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.504766 25741 raft_consensus.cc:2069] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.504607 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:49.505029 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:49.505250 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:49.505496 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:49.505959 25729 raft_consensus.cc:457] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.505957 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 6d6363f6403b4104b15b329ab7f0e483 (table default.table_7ea77d77ddd04410989cd2735133a5e5 [id=dd175630a9ac4af5aa17edcb7226e141]) successfully deleted
I0619 00:11:49.506184 25729 raft_consensus.cc:2732] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.913s (starting election)
I0619 00:11:49.506204 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 60dfd91c98d94bfbbc6416268c65c378 (table default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]) successfully deleted
W0619 00:11:49.506456 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet baf80fd968c44e679987eb4ebe4473de with error code TABLET_NOT_RUNNING: Already present: State transition of tablet baf80fd968c44e679987eb4ebe4473de already in progress: creating tablet
I0619 00:11:49.506667 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 165 ms (attempt = 4)
I0619 00:11:49.506412 25729 raft_consensus.cc:479] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.507313 25336 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.507375 25729 leader_election.cc:243] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.507733 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.507774 25729 leader_election.cc:243] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:49.507961 25727 tablet_bootstrap.cc:427] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:49.508267 25727 ts_tablet_manager.cc:1045] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.096s	user 0.004s	sys 0.000s
W0619 00:11:49.509658 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 because the tablet was not found. No further retry: Not found: Tablet not found: 6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:49.509608 25727 raft_consensus.cc:325] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.510202 25741 ts_tablet_manager.cc:1369] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:49.510380 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:49.510593 25067 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 98 ms (attempt = 3)
I0619 00:11:49.510207 25727 raft_consensus.cc:351] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.511296 25727 raft_consensus.cc:628] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.512531 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:49.511739 25727 consensus_queue.cc:227] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.512857 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:49.513062 25730 raft_consensus.cc:2042] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.513347 25730 raft_consensus.cc:2069] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.514497 25727 ts_tablet_manager.cc:1073] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I0619 00:11:49.514830 25727 ts_tablet_manager.cc:1028] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:49.515995 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.515992 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1d10fb4561924e2e8d53c7a0b22a426e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:49.515184 25727 tablet_bootstrap.cc:427] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:49.515663 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6d6363f6403b4104b15b329ab7f0e483 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
W0619 00:11:49.517330 25558 leader_election.cc:306] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.517498 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1d10fb4561924e2e8d53c7a0b22a426e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:49.518296 25069 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 64 ms (attempt = 2)
W0619 00:11:49.518556 25069 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
W0619 00:11:49.519012 25553 leader_election.cc:306] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:49.519116 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.519259 25553 leader_election.cc:270] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.520596 25729 raft_consensus.cc:2732] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.787s (election complete)
I0619 00:11:49.520839 25729 raft_consensus.cc:2487] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.522094 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 6d6363f6403b4104b15b329ab7f0e483 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.522905 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.523090 25725 tablet_bootstrap.cc:427] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.523427 25725 ts_tablet_manager.cc:1045] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.077s	user 0.003s	sys 0.007s
I0619 00:11:49.523978 25604 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.524085 25067 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 72 ms (attempt = 2)
W0619 00:11:49.524441 25067 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.524328 25083 catalog_manager.cc:3663] Got report from deleted tablet 60dfd91c98d94bfbbc6416268c65c378 (table default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:49.526293 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 60dfd91c98d94bfbbc6416268c65c378 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.528704 25727 tablet_bootstrap.cc:578] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.531925 25730 ts_tablet_manager.cc:1369] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.531765 25469 tablet_service.cc:759] Processing CreateTablet for tablet 35acf55f35064c1491122a13ff9f5744 (table=default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.533079 25736 ts_tablet_manager.cc:1382] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.533321 25736 log.cc:986] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:49.533476 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 35acf55f35064c1491122a13ff9f5744. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.532889 25604 tablet_service.cc:759] Processing CreateTablet for tablet 35acf55f35064c1491122a13ff9f5744 (table=default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.533840 25736 ts_tablet_manager.cc:1403] T 89a7ff9cdbc046779799e4e8f71adb42 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:49.534078 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 35acf55f35064c1491122a13ff9f5744. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.536206 25736 tablet_replica.cc:261] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:49.536427 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.536643 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.536799 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) successfully deleted
W0619 00:11:49.537272 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet dc04126054714196af6bdf7e771ffab0 because the tablet was not found. No further retry: Not found: Tablet not found: dc04126054714196af6bdf7e771ffab0
I0619 00:11:49.538331 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.533481 25724 tablet_bootstrap.cc:427] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.538887 25724 ts_tablet_manager.cc:1045] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.048s	user 0.004s	sys 0.001s
I0619 00:11:49.539288 25326 tablet_service.cc:759] Processing CreateTablet for tablet 35acf55f35064c1491122a13ff9f5744 (table=default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.539875 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 35acf55f35064c1491122a13ff9f5744. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.540779 25724 raft_consensus.cc:325] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.543133 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 256 ms (attempt = 5)
W0619 00:11:49.543403 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.543776 25724 raft_consensus.cc:351] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.543987 25724 raft_consensus.cc:628] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:49.543468 25743 raft_consensus.cc:457] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.545867 25743 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.798s (starting election)
I0619 00:11:49.546222 25743 raft_consensus.cc:479] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.547504 25743 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.548153 25743 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:49.548856 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 171 ms (attempt = 4)
W0619 00:11:49.549123 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.545843 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.546921 25724 consensus_queue.cc:227] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.553920 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.554378 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.554044 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_0a2ef368d0f945eaaeeb3e58d57ef718"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:49.554929 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(baf80fd968c44e679987eb4ebe4473de)
W0619 00:11:49.554961 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.555371 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.555779 25736 raft_consensus.cc:2042] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.555877 25724 ts_tablet_manager.cc:1073] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.001s
I0619 00:11:49.556862 25741 ts_tablet_manager.cc:1382] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.557189 25736 raft_consensus.cc:2069] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.558907 25736 ts_tablet_manager.cc:1369] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.558801 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_514667669c72462a95c41b6bb68dc0fe" } new_table_name: "default.table_5ee63bff24ae439588b05c825c6c9282" alter_external_catalogs: true
I0619 00:11:49.559500 25741 log.cc:986] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a226a7e97a7e44af8301ed4e6363effb
I0619 00:11:49.559545 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_80d7e883b20847feb66d8a68bae7d094"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:49.560287 25741 ts_tablet_manager.cc:1403] T a226a7e97a7e44af8301ed4e6363effb P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:49.560556 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_80d7e883b20847feb66d8a68bae7d094 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.546458 25604 ts_tablet_manager.cc:1173] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.558991 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 289 ms (attempt = 5)
W0619 00:11:49.564131 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_514667669c72462a95c41b6bb68dc0fe [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.569305 25087 catalog_manager.cc:3663] Got report from deleted tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:49.570031 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:49.570843 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times.
I0619 00:11:49.571921 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.572494 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_b4370a7e80854fd8965e81d09a545d89" } new_table_name: "default.table_31a3ade4f34e4e7a96d0bc91603e471d" alter_external_catalogs: true
I0619 00:11:49.573740 25724 ts_tablet_manager.cc:1028] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.575120 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet a226a7e97a7e44af8301ed4e6363effb (table default.table_de92feb7880c4ea2b3fd76c65c6b3df5 [id=122dad4ddc134d21b827dca8587ed257]) successfully deleted
I0619 00:11:49.575251 25724 tablet_bootstrap.cc:427] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.576206 25080 catalog_manager.cc:3663] Got report from deleted tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:49.576263 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 167 ms (attempt = 4)
W0619 00:11:49.576668 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.576858 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:49.577015 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a226a7e97a7e44af8301ed4e6363effb because the tablet was not found. No further retry: Not found: Tablet not found: a226a7e97a7e44af8301ed4e6363effb
I0619 00:11:49.577400 25083 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 46 ms (attempt = 1)
W0619 00:11:49.577718 25083 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.544934 25725 raft_consensus.cc:325] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.578203 25725 raft_consensus.cc:351] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.578419 25725 raft_consensus.cc:628] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.579329 25463 tablet_service.cc:759] Processing CreateTablet for tablet acc89522b693498c987ca5d8ef8275ea (table=default.table_b4370a7e80854fd8965e81d09a545d89 [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.579833 25604 tablet_service.cc:759] Processing CreateTablet for tablet acc89522b693498c987ca5d8ef8275ea (table=default.table_b4370a7e80854fd8965e81d09a545d89 [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.579427 25725 consensus_queue.cc:227] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.580303 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 299 ms (attempt = 5)
W0619 00:11:49.580626 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.580561 25467 tablet_service.cc:759] Processing CreateTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 (table=default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.581220 25725 ts_tablet_manager.cc:1073] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.037s	user 0.002s	sys 0.001s
I0619 00:11:49.582365 25326 ts_tablet_manager.cc:1173] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.565971 25469 ts_tablet_manager.cc:1173] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.583124 25741 tablet_replica.cc:261] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:49.583561 25069 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 77 ms (attempt = 3)
W0619 00:11:49.583830 25069 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.584200 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet acc89522b693498c987ca5d8ef8275ea. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.583160 25603 tablet_service.cc:759] Processing CreateTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 (table=default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.584676 25067 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 132 ms (attempt = 4)
I0619 00:11:49.584861 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 70b9c8c4af0945b69b5577e47b61c054. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.588893 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 70b9c8c4af0945b69b5577e47b61c054. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.589188 25067 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.588906 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.589673 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.590858 25336 tablet_service.cc:759] Processing CreateTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 (table=default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.591650 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 70b9c8c4af0945b69b5577e47b61c054. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.595983 25603 ts_tablet_manager.cc:1173] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.597126 25326 tablet_service.cc:759] Processing CreateTablet for tablet acc89522b693498c987ca5d8ef8275ea (table=default.table_b4370a7e80854fd8965e81d09a545d89 [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.597720 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet acc89522b693498c987ca5d8ef8275ea. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.598152 25467 ts_tablet_manager.cc:1173] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.602592 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.603396 25067 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 96 ms (attempt = 3)
W0619 00:11:49.604285 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.604779 25067 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.598142 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet acc89522b693498c987ca5d8ef8275ea. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.606290 25463 ts_tablet_manager.cc:1173] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.606727 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a226a7e97a7e44af8301ed4e6363effb because the tablet was not found. No further retry: Not found: Tablet not found: a226a7e97a7e44af8301ed4e6363effb
I0619 00:11:49.608520 25729 raft_consensus.cc:457] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.608785 25729 raft_consensus.cc:2732] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.905s (starting election)
I0619 00:11:49.609035 25729 raft_consensus.cc:479] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.609536 25083 catalog_manager.cc:3663] Got report from deleted tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.610029 25729 leader_election.cc:243] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.610435 25729 leader_election.cc:243] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:49.610437 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
W0619 00:11:49.612432 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.614665 25725 ts_tablet_manager.cc:1028] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.614822 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4c0614e9ed96420886243a6868f2e75e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:49.615061 25725 tablet_bootstrap.cc:427] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:49.615517 25724 tablet_bootstrap.cc:578] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:49.617044 25558 leader_election.cc:306] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.618176 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.618418 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.618955 25463 tablet_service.cc:759] Processing CreateTablet for tablet cb57d196f0724247b6a7f9a34be8377a (table=default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.619529 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cb57d196f0724247b6a7f9a34be8377a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.620306 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.623273 25087 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 60 ms (attempt = 1)
W0619 00:11:49.623569 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:49.623977 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 55 ms (attempt = 2)
W0619 00:11:49.624243 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.627730 25603 tablet_service.cc:759] Processing CreateTablet for tablet cb57d196f0724247b6a7f9a34be8377a (table=default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.628338 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cb57d196f0724247b6a7f9a34be8377a. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.629403 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.601032 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.602862 25325 tablet_service.cc:759] Processing CreateTablet for tablet cb57d196f0724247b6a7f9a34be8377a (table=default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.631844 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.632128 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.600549 25336 ts_tablet_manager.cc:1173] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.632371 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.632454 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4c0614e9ed96420886243a6868f2e75e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:49.635901 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:49.632823 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cb57d196f0724247b6a7f9a34be8377a. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.636817 25553 leader_election.cc:306] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.637104 25553 leader_election.cc:270] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.637346 25736 ts_tablet_manager.cc:1382] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.637547 25729 raft_consensus.cc:2732] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.577s (election complete)
I0619 00:11:49.637743 25725 tablet_bootstrap.cc:578] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.637789 25729 raft_consensus.cc:2487] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.637931 25741 raft_consensus.cc:2042] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.638195 25741 raft_consensus.cc:2069] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.638911 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:49.639956 25624 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.581s (vote granted)
I0619 00:11:49.640192 25624 raft_consensus.cc:2245] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:49.640795 25741 ts_tablet_manager.cc:1369] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.641520 25423 leader_election.cc:379] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:49.641842 25423 leader_election.cc:270] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:49.642416 25743 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.736s (election complete)
I0619 00:11:49.642691 25743 raft_consensus.cc:2539] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:49.642923 25743 raft_consensus.cc:457] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:49.643126 25743 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.855s (starting election)
I0619 00:11:49.643332 25743 raft_consensus.cc:2784] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:49.645752 25087 catalog_manager.cc:3663] Got report from deleted tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:49.646456 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:49.647977 25336 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.648864 25743 raft_consensus.cc:479] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.649891 25743 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.650826 25743 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:49.652982 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:49.653411 25624 raft_consensus.cc:2784] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:49.653702 25624 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.831s (vote granted)
I0619 00:11:49.654901 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.655124 25729 raft_consensus.cc:457] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.655355 25729 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.017s (starting election)
I0619 00:11:49.655592 25729 raft_consensus.cc:479] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.656508 25729 leader_election.cc:243] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:49.656893 25729 leader_election.cc:243] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.657371 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_f4fd4524d920428fa515c1266d90fb2a" }
I0619 00:11:49.658990 25624 raft_consensus.cc:2245] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:49.661088 25423 leader_election.cc:379] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:49.661027 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fb8e94875a8347a0a473c95a77c38795" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:49.661425 25423 leader_election.cc:270] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:49.662200 25069 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC with a delay of 166 ms (attempt = 4)
W0619 00:11:49.662447 25069 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
W0619 00:11:49.663221 25558 leader_election.cc:306] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.663738 25743 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.916s (election complete)
I0619 00:11:49.637575 25736 log.cc:986] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/baf80fd968c44e679987eb4ebe4473de
I0619 00:11:49.664898 25736 ts_tablet_manager.cc:1403] T baf80fd968c44e679987eb4ebe4473de P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:49.667619 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) successfully deleted
W0619 00:11:49.668381 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 because the tablet was not found. No further retry: Not found: Tablet not found: 89a7ff9cdbc046779799e4e8f71adb42
W0619 00:11:49.669709 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 89a7ff9cdbc046779799e4e8f71adb42 because the tablet was not found. No further retry: Not found: Tablet not found: 89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:49.664014 25743 raft_consensus.cc:2539] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:49.671392 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 1d10fb4561924e2e8d53c7a0b22a426e
I0619 00:11:49.671442 25467 tablet_service.cc:759] Processing CreateTablet for tablet 03522c48c50943b38cae5aa88562e583 (table=default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.671793 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.672547 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.672863 25601 tablet_service.cc:759] Processing CreateTablet for tablet 03522c48c50943b38cae5aa88562e583 (table=default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.673099 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.673211 25602 tablet_service.cc:759] Processing CreateTablet for tablet f372b9fb174844eca1662c62039a3c71 (table=default.table_80d7e883b20847feb66d8a68bae7d094 [id=60a6d49e09f04230abe707ed433d2e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.673336 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.673508 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 03522c48c50943b38cae5aa88562e583. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.673828 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f372b9fb174844eca1662c62039a3c71. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.675050 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.675370 25336 tablet_service.cc:759] Processing CreateTablet for tablet 03522c48c50943b38cae5aa88562e583 (table=default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.675453 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:49.676086 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 03522c48c50943b38cae5aa88562e583. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.651504 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:49.676554 25727 tablet_bootstrap.cc:427] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:49.676893 25727 ts_tablet_manager.cc:1045] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.162s	user 0.008s	sys 0.002s
I0619 00:11:49.677398 25330 tablet_service.cc:759] Processing CreateTablet for tablet f372b9fb174844eca1662c62039a3c71 (table=default.table_80d7e883b20847feb66d8a68bae7d094 [id=60a6d49e09f04230abe707ed433d2e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.677929 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f372b9fb174844eca1662c62039a3c71. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.678354 25333 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.678305 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:49.678416 25419 leader_election.cc:306] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:49.679386 25419 leader_election.cc:306] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.680045 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 65 ms (attempt = 3)
W0619 00:11:49.680317 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.684535 25730 ts_tablet_manager.cc:1382] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.684830 25730 log.cc:986] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/ac129ed44f68432f800420175aa0a630
I0619 00:11:49.686400 25730 ts_tablet_manager.cc:1403] T ac129ed44f68432f800420175aa0a630 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:49.689157 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 33 ms (attempt = 2)
W0619 00:11:49.689458 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:49.689784 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet ac129ed44f68432f800420175aa0a630 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) successfully deleted
I0619 00:11:49.690068 25743 raft_consensus.cc:593] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:49.690436 25602 ts_tablet_manager.cc:1173] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.689707 25326 ts_tablet_manager.cc:1173] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.691391 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 03522c48c50943b38cae5aa88562e583. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.690969 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fb8e94875a8347a0a473c95a77c38795" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:49.694411 25553 leader_election.cc:306] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.694705 25553 leader_election.cc:270] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.695385 25729 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.033s (election complete)
I0619 00:11:49.696578 25727 raft_consensus.cc:325] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.697190 25727 raft_consensus.cc:351] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.697379 25727 raft_consensus.cc:628] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.697804 25727 consensus_queue.cc:227] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.699081 25724 tablet_bootstrap.cc:427] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.699406 25724 ts_tablet_manager.cc:1045] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.124s	user 0.004s	sys 0.003s
W0619 00:11:49.703925 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.704191 25067 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.704068 25724 raft_consensus.cc:325] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.704712 25724 raft_consensus.cc:351] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.704916 25724 raft_consensus.cc:628] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:49.705570 25727 ts_tablet_manager.cc:1073] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.028s	user 0.003s	sys 0.001s
I0619 00:11:49.705934 25727 ts_tablet_manager.cc:1028] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:49.705443 25724 consensus_queue.cc:227] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.706281 25727 tablet_bootstrap.cc:427] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:49.695583 25604 ts_tablet_manager.cc:1173] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.708546 25727 tablet_bootstrap.cc:578] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.708845 25601 ts_tablet_manager.cc:1173] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.708904 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 535 ms (attempt = 6)
W0619 00:11:49.709249 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:49.695595 25729 raft_consensus.cc:2487] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.714035 25083 master_service.cc:523] ReplaceTablet: received request to replace tablet 70b9c8c4af0945b69b5577e47b61c054 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:49.714534 25082 master_service.cc:523] ReplaceTablet: received request to replace tablet 292b164659e343158de8e7e3eeb9be31 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:49.709662 25325 ts_tablet_manager.cc:1173] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.717185 25330 ts_tablet_manager.cc:1173] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.712932 25741 ts_tablet_manager.cc:1382] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.717798 25741 log.cc:986] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/60dfd91c98d94bfbbc6416268c65c378
W0619 00:11:49.712916 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.718272 25741 ts_tablet_manager.cc:1403] T 60dfd91c98d94bfbbc6416268c65c378 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:49.709262 25087 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 2 got 0
I0619 00:11:49.722093 25087 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 31 ms (attempt = 1)
I0619 00:11:49.722093 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 267 ms (attempt = 5)
W0619 00:11:49.722399 25087 catalog_manager.cc:3945] Failed to send e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
W0619 00:11:49.722486 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.724020 25730 tablet_replica.cc:261] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:49.724328 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:49.724567 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(4c0614e9ed96420886243a6868f2e75e)
W0619 00:11:49.722162 25067 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.724838 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:49.725069 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:49.725096 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 60dfd91c98d94bfbbc6416268c65c378 (table default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]) successfully deleted
I0619 00:11:49.725350 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(4c0614e9ed96420886243a6868f2e75e)
W0619 00:11:49.725559 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.725559 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:49.726050 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:49.726227 25730 raft_consensus.cc:2042] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.726522 25730 raft_consensus.cc:2069] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:49.726579 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:49.726769 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 70b9c8c4af0945b69b5577e47b61c054
I0619 00:11:49.727761 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:49.728181 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:49.729194 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:49.729468 25603 ts_tablet_manager.cc:1173] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.730593 25736 tablet_replica.cc:261] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:49.730913 25083 catalog_manager.cc:4596] ReplaceTablet: tablet 70b9c8c4af0945b69b5577e47b61c054 deleted and replaced by tablet a85577b0beba4276b963e5c360544bc7
I0619 00:11:49.730903 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:49.731132 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:49.726821 25067 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 47 ms (attempt = 1)
I0619 00:11:49.732048 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 77 ms (attempt = 3)
I0619 00:11:49.737686 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:49.737951 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:49.738219 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:49.738472 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:49.738718 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:49.738859 25336 ts_tablet_manager.cc:1173] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.737421 25463 ts_tablet_manager.cc:1173] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.736191 25743 consensus_queue.cc:204] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:49.739756 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:49.734777 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 292b164659e343158de8e7e3eeb9be31
W0619 00:11:49.740061 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:49.740088 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:49.740293 25071 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 49 ms (attempt = 1)
I0619 00:11:49.740504 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:49.741503 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:49.741868 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_62055e93837844cd9a5dc2a343a23878" }
I0619 00:11:49.742094 25082 catalog_manager.cc:4596] ReplaceTablet: tablet 292b164659e343158de8e7e3eeb9be31 deleted and replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0
W0619 00:11:49.732154 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet ac129ed44f68432f800420175aa0a630 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ac129ed44f68432f800420175aa0a630 already in progress: creating tablet
I0619 00:11:49.742195 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_50bf60a48eb949a2b7c22f6b61030f66"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:49.742655 25067 catalog_manager.cc:2891] Scheduling retry of ac129ed44f68432f800420175aa0a630 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 65 ms (attempt = 1)
I0619 00:11:49.742776 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:49.743118 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_50bf60a48eb949a2b7c22f6b61030f66 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:49.743428 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:49.743676 25067 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
W0619 00:11:49.744151 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 6d6363f6403b4104b15b329ab7f0e483 because the tablet was not found. No further retry: Not found: Tablet not found: 6d6363f6403b4104b15b329ab7f0e483
I0619 00:11:49.744726 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 291 ms (attempt = 5)
W0619 00:11:49.744999 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
W0619 00:11:49.745352 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 60dfd91c98d94bfbbc6416268c65c378 because the tablet was not found. No further retry: Not found: Tablet not found: 60dfd91c98d94bfbbc6416268c65c378
I0619 00:11:49.745625 25730 ts_tablet_manager.cc:1369] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:49.745854 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:49.746063 25067 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 142 ms (attempt = 4)
I0619 00:11:49.744607 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.745009 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.746474 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 142 ms (attempt = 4)
W0619 00:11:49.746794 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.747543 25725 tablet_bootstrap.cc:427] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.747905 25725 ts_tablet_manager.cc:1045] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.133s	user 0.006s	sys 0.007s
I0619 00:11:49.747618 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_6eabaa2225f44436880ea396f6bc6acd"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:49.748456 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6eabaa2225f44436880ea396f6bc6acd in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:49.749362 25725 raft_consensus.cc:325] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.750006 25725 raft_consensus.cc:351] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.750278 25725 raft_consensus.cc:628] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.752259 25727 tablet_bootstrap.cc:427] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:49.752648 25727 ts_tablet_manager.cc:1045] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.046s	user 0.004s	sys 0.002s
I0619 00:11:49.738920 25736 raft_consensus.cc:2042] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.753355 25736 raft_consensus.cc:2069] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.754055 25727 raft_consensus.cc:325] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.754729 25727 raft_consensus.cc:351] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.754941 25727 raft_consensus.cc:628] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.755307 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.755532 25727 consensus_queue.cc:227] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.756206 25463 tablet_service.cc:759] Processing CreateTablet for tablet f372b9fb174844eca1662c62039a3c71 (table=default.table_80d7e883b20847feb66d8a68bae7d094 [id=60a6d49e09f04230abe707ed433d2e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.756819 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f372b9fb174844eca1662c62039a3c71. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.758600 25409 tablet.cc:1678] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0619 00:11:49.760726 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.763465 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.764449 25736 ts_tablet_manager.cc:1369] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.766106 25729 raft_consensus.cc:457] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.766356 25729 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.891s (starting election)
I0619 00:11:49.766595 25729 raft_consensus.cc:479] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.767570 25729 leader_election.cc:243] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.767973 25729 leader_election.cc:243] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:49.746502 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet baf80fd968c44e679987eb4ebe4473de with error code TABLET_NOT_RUNNING: Already present: State transition of tablet baf80fd968c44e679987eb4ebe4473de already in progress: creating tablet
I0619 00:11:49.769953 25067 catalog_manager.cc:2891] Scheduling retry of baf80fd968c44e679987eb4ebe4473de Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 275 ms (attempt = 5)
I0619 00:11:49.769881 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72e28bc6b57442b782d8aa07e1152cf4" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:49.770582 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 78 ms (attempt = 2)
W0619 00:11:49.770895 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
W0619 00:11:49.771405 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 1d10fb4561924e2e8d53c7a0b22a426e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 1d10fb4561924e2e8d53c7a0b22a426e already in progress: creating tablet
W0619 00:11:49.771380 25553 leader_election.cc:306] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.771672 25067 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 23 ms (attempt = 1)
I0619 00:11:49.771971 25467 ts_tablet_manager.cc:1173] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.772783 25727 ts_tablet_manager.cc:1073] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I0619 00:11:49.773171 25727 ts_tablet_manager.cc:1028] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:49.773574 25080 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:49.774307 25080 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 38 ms (attempt = 1)
I0619 00:11:49.774528 25727 tablet_bootstrap.cc:427] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:49.774567 25080 catalog_manager.cc:3945] Failed to send fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.776973 25727 tablet_bootstrap.cc:578] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.775862 25725 consensus_queue.cc:227] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.778654 25725 ts_tablet_manager.cc:1073] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.030s	user 0.004s	sys 0.000s
I0619 00:11:49.779007 25725 ts_tablet_manager.cc:1028] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.779783 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:49.780074 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.780460 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.780861 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:49.779268 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:49.781265 25336 tablet_service.cc:759] Processing CreateTablet for tablet a85577b0beba4276b963e5c360544bc7 (table=default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.781388 25071 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 72 ms (attempt = 2)
I0619 00:11:49.781913 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a85577b0beba4276b963e5c360544bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.782251 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3d4be1da5dd54e91b7a565ceded2dae3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:49.783123 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3d4be1da5dd54e91b7a565ceded2dae3 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:49.785114 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet baf80fd968c44e679987eb4ebe4473de because the tablet was not found. No further retry: Not found: Tablet not found: baf80fd968c44e679987eb4ebe4473de
I0619 00:11:49.785807 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:49.786280 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet dc04126054714196af6bdf7e771ffab0 because the tablet was not found. No further retry: Not found: Tablet not found: dc04126054714196af6bdf7e771ffab0
I0619 00:11:49.786824 25601 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.787981 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.791029 25725 tablet_bootstrap.cc:427] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:49.789147 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.789522 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72e28bc6b57442b782d8aa07e1152cf4" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:49.789918 25462 tablet_service.cc:759] Processing CreateTablet for tablet a85577b0beba4276b963e5c360544bc7 (table=default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:49.788847 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.791993 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a85577b0beba4276b963e5c360544bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.802476 25336 ts_tablet_manager.cc:1173] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.803357 25725 tablet_bootstrap.cc:578] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.805065 25724 ts_tablet_manager.cc:1073] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.105s	user 0.005s	sys 0.001s
I0619 00:11:49.805407 25724 ts_tablet_manager.cc:1028] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.805788 25724 tablet_bootstrap.cc:427] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.808137 25724 tablet_bootstrap.cc:578] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:49.810958 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.811278 25558 leader_election.cc:306] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.811545 25558 leader_election.cc:270] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.815151 25776 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.802s (election complete)
I0619 00:11:49.815402 25776 raft_consensus.cc:2487] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.817019 25604 tablet_service.cc:759] Processing CreateTablet for tablet a85577b0beba4276b963e5c360544bc7 (table=default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.817730 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a85577b0beba4276b963e5c360544bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.789901 25080 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_80d7e883b20847feb66d8a68bae7d094" } new_table_name: "default.table_45dd93d67cc94f2c8e9e784347d14167" alter_external_catalogs: true
I0619 00:11:49.820559 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:49.800513 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 523 ms (attempt = 6)
W0619 00:11:49.820863 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.821403 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 55 ms (attempt = 2)
W0619 00:11:49.821676 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.795253 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.821995 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 165 ms (attempt = 4)
W0619 00:11:49.822237 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:49.822428 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
W0619 00:11:49.822790 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 60dfd91c98d94bfbbc6416268c65c378 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 60dfd91c98d94bfbbc6416268c65c378 already in progress: creating tablet
I0619 00:11:49.823084 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.823194 25730 ts_tablet_manager.cc:1382] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.823004 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_0a0cd847ea574c2eb82ae9dfaf67e95f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:49.823493 25730 log.cc:986] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/4c0614e9ed96420886243a6868f2e75e
W0619 00:11:49.823973 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0a0cd847ea574c2eb82ae9dfaf67e95f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:49.823904 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:49.824156 25730 ts_tablet_manager.cc:1403] T 4c0614e9ed96420886243a6868f2e75e P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:49.824225 25067 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 52 ms (attempt = 1)
I0619 00:11:49.825620 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
W0619 00:11:49.829614 25069 catalog_manager.cc:2910] Async tablet task 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d10fb4561924e2e8d53c7a0b22a426e consensus state has no leader
I0619 00:11:49.829802 25727 tablet_bootstrap.cc:427] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
W0619 00:11:49.811398 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.824282 25467 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.830274 25727 ts_tablet_manager.cc:1045] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.057s	user 0.005s	sys 0.001s
I0619 00:11:49.831755 25727 raft_consensus.cc:325] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.832438 25727 raft_consensus.cc:351] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.832670 25727 raft_consensus.cc:628] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.823019 25071 catalog_manager.cc:2891] Scheduling retry of 60dfd91c98d94bfbbc6416268c65c378 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 23 ms (attempt = 1)
I0619 00:11:49.833505 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) successfully deleted
I0619 00:11:49.833251 25727 consensus_queue.cc:227] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.835057 25601 tablet_service.cc:759] Processing CreateTablet for tablet c18355df78d344839a56189c7549ab1f (table=default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:49.835937 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.836248 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c18355df78d344839a56189c7549ab1f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.836436 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.837093 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_5ee63bff24ae439588b05c825c6c9282" } new_table_name: "default.table_6ace57434b5243c7bd22bd60752a065d" alter_external_catalogs: true
W0619 00:11:49.838518 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.835461 25603 tablet_service.cc:759] Processing CreateTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table=default.table_aecb1278dc98420aa83aaae47fb56937 [id=e2a7900f428948bab0aefd132d8874d4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.840176 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ee4c4769f75141f9b2ef864ab8a8d9e0. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.841872 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet ac129ed44f68432f800420175aa0a630 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ac129ed44f68432f800420175aa0a630 already in progress: creating tablet
I0619 00:11:49.842247 25071 catalog_manager.cc:2891] Scheduling retry of ac129ed44f68432f800420175aa0a630 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 59 ms (attempt = 1)
I0619 00:11:49.843369 25467 tablet_service.cc:759] Processing CreateTablet for tablet c18355df78d344839a56189c7549ab1f (table=default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:49.847774 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 70b9c8c4af0945b69b5577e47b61c054 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 70b9c8c4af0945b69b5577e47b61c054 already in progress: creating tablet
I0619 00:11:49.848033 25067 catalog_manager.cc:2891] Scheduling retry of 70b9c8c4af0945b69b5577e47b61c054 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 55 ms (attempt = 1)
I0619 00:11:49.826292 25463 ts_tablet_manager.cc:1173] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.849313 25462 ts_tablet_manager.cc:1173] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.849314 25330 tablet_service.cc:759] Processing CreateTablet for tablet c18355df78d344839a56189c7549ab1f (table=default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.849874 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c18355df78d344839a56189c7549ab1f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.849937 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c18355df78d344839a56189c7549ab1f. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:49.851819 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e because the tablet was not found. No further retry: Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
W0619 00:11:49.852227 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:49.852432 25071 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 47 ms (attempt = 1)
W0619 00:11:49.852823 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e because the tablet was not found. No further retry: Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
W0619 00:11:49.853206 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:49.853835 25071 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 36 ms (attempt = 1)
I0619 00:11:49.855034 25462 tablet_service.cc:759] Processing CreateTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table=default.table_aecb1278dc98420aa83aaae47fb56937 [id=e2a7900f428948bab0aefd132d8874d4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.855684 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ee4c4769f75141f9b2ef864ab8a8d9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.857421 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.857558 25325 tablet_service.cc:759] Processing CreateTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table=default.table_aecb1278dc98420aa83aaae47fb56937 [id=e2a7900f428948bab0aefd132d8874d4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.857678 25080 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 30 ms (attempt = 1)
W0619 00:11:49.857945 25080 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:49.858465 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ee4c4769f75141f9b2ef864ab8a8d9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.858889 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 85 ms (attempt = 3)
W0619 00:11:49.859163 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_5ee63bff24ae439588b05c825c6c9282 [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.859516 25336 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.862324 25330 ts_tablet_manager.cc:1173] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.862859 25082 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 34 ms (attempt = 1)
W0619 00:11:49.863129 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.863709 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.864310 25724 tablet_bootstrap.cc:427] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:49.864567 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 546 ms (attempt = 6)
I0619 00:11:49.864689 25724 ts_tablet_manager.cc:1045] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.059s	user 0.006s	sys 0.000s
W0619 00:11:49.864902 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
W0619 00:11:49.865972 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:49.866724 25724 raft_consensus.cc:325] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.867341 25724 raft_consensus.cc:351] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.867525 25724 raft_consensus.cc:628] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:49.868024 25724 consensus_queue.cc:227] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.869325 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.870405 25067 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 65 ms (attempt = 2)
W0619 00:11:49.870715 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.833869 25730 tablet_replica.cc:261] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:49.871961 25330 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.873325 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:49.873558 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:49.873908 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:49.874125 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:49.875190 25325 ts_tablet_manager.cc:1173] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.835232 25727 ts_tablet_manager.cc:1073] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I0619 00:11:49.876101 25727 ts_tablet_manager.cc:1028] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:49.876466 25727 tablet_bootstrap.cc:427] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:49.876441 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:49.878970 25727 tablet_bootstrap.cc:578] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:49.879712 25067 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 30 ms (attempt = 1)
I0619 00:11:49.880089 25467 ts_tablet_manager.cc:1173] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.880244 25736 ts_tablet_manager.cc:1382] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.881214 25736 log.cc:986] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/985e23a32fa84104a06dd19d97117ab2
I0619 00:11:49.881772 25736 ts_tablet_manager.cc:1403] T 985e23a32fa84104a06dd19d97117ab2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:49.886060 25725 tablet_bootstrap.cc:427] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:49.886411 25725 ts_tablet_manager.cc:1045] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.095s	user 0.003s	sys 0.004s
I0619 00:11:49.887889 25725 raft_consensus.cc:325] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.888509 25725 raft_consensus.cc:351] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.888751 25725 raft_consensus.cc:628] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:49.889298 25725 consensus_queue.cc:227] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:49.894253 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:49.896104 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.897399 25725 ts_tablet_manager.cc:1073] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.000s
I0619 00:11:49.898842 25725 ts_tablet_manager.cc:1028] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:49.899220 25725 tablet_bootstrap.cc:427] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:49.880689 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 529 ms (attempt = 6)
I0619 00:11:49.886086 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 60dfd91c98d94bfbbc6416268c65c378 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.891057 25082 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:49.893239 25083 catalog_manager.cc:3786] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
W0619 00:11:49.901337 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.901818 25069 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 303 ms (attempt = 5)
I0619 00:11:49.901897 25082 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 16 ms (attempt = 1)
I0619 00:11:49.902055 25083 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 3 got 0
W0619 00:11:49.902165 25082 catalog_manager.cc:3945] Failed to send fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
W0619 00:11:49.902060 25069 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
I0619 00:11:49.902928 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.903146 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:49.903714 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_d8b3984991a442fe95208294f57d466f" }
I0619 00:11:49.882196 25724 ts_tablet_manager.cc:1073] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.001s
W0619 00:11:49.885480 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.905730 25724 ts_tablet_manager.cc:1028] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:49.906100 25724 tablet_bootstrap.cc:427] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:49.906512 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 88 ms (attempt = 3)
W0619 00:11:49.907002 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.907512 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:49.908192 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.903668 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.910516 25067 catalog_manager.cc:2891] Scheduling retry of e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 67 ms (attempt = 2)
W0619 00:11:49.910820 25067 catalog_manager.cc:2910] Async tablet task e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e3708e05f6054306b5bf7f1df2140e3d consensus state has no leader
W0619 00:11:49.912513 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 1d10fb4561924e2e8d53c7a0b22a426e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 1d10fb4561924e2e8d53c7a0b22a426e already in progress: creating tablet
I0619 00:11:49.912781 25067 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 65 ms (attempt = 2)
I0619 00:11:49.914173 25330 tablet_service.cc:759] Processing CreateTablet for tablet 397ae10bb6d841b4ba561ec431d29864 (table=default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.914824 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 397ae10bb6d841b4ba561ec431d29864. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.917187 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:49.917707 25724 tablet_bootstrap.cc:578] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:49.920274 25325 tablet_service.cc:759] Processing CreateTablet for tablet 3722b203a516466489c74ef6013eacae (table=default.table_6eabaa2225f44436880ea396f6bc6acd [id=a17737f9d2924cc09f372527fb91188b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.920368 25603 ts_tablet_manager.cc:1173] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.920379 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.920886 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3722b203a516466489c74ef6013eacae. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.921785 25336 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.919147 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:49.922467 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:49.922662 25730 raft_consensus.cc:2042] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.922876 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:49.922915 25730 raft_consensus.cc:2069] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:49.923437 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:49.923691 25067 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 72 ms (attempt = 2)
W0619 00:11:49.925416 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet ac129ed44f68432f800420175aa0a630 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ac129ed44f68432f800420175aa0a630 already in progress: creating tablet
I0619 00:11:49.925690 25067 catalog_manager.cc:2891] Scheduling retry of ac129ed44f68432f800420175aa0a630 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 39 ms (attempt = 2)
I0619 00:11:49.926005 25087 catalog_manager.cc:3663] Got report from deleted tablet baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:49.926668 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.926725 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:49.927862 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:49.929790 25743 raft_consensus.cc:457] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.930043 25743 raft_consensus.cc:2732] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.701s (starting election)
I0619 00:11:49.930603 25602 tablet_service.cc:759] Processing CreateTablet for tablet 3722b203a516466489c74ef6013eacae (table=default.table_6eabaa2225f44436880ea396f6bc6acd [id=a17737f9d2924cc09f372527fb91188b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.930290 25743 raft_consensus.cc:479] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.931211 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3722b203a516466489c74ef6013eacae. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.931293 25743 leader_election.cc:243] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:49.931711 25743 leader_election.cc:243] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:49.932859 25730 ts_tablet_manager.cc:1369] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.934079 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ac129ed44f68432f800420175aa0a630" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:49.935465 25423 leader_election.cc:306] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: ac129ed44f68432f800420175aa0a630
W0619 00:11:49.936534 25419 leader_election.cc:306] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:49.936815 25419 leader_election.cc:270] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:49.936929 25603 tablet_service.cc:759] Processing CreateTablet for tablet 397ae10bb6d841b4ba561ec431d29864 (table=default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.937481 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 397ae10bb6d841b4ba561ec431d29864. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.937440 25743 raft_consensus.cc:2732] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.087s (election complete)
I0619 00:11:49.937682 25743 raft_consensus.cc:2487] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:49.938067 25725 tablet_bootstrap.cc:578] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:49.938246 25325 ts_tablet_manager.cc:1173] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.939193 25462 ts_tablet_manager.cc:1173] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.940384 25604 ts_tablet_manager.cc:1173] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.941980 25727 tablet_bootstrap.cc:427] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:49.942303 25727 ts_tablet_manager.cc:1045] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.066s	user 0.005s	sys 0.000s
I0619 00:11:49.943625 25727 raft_consensus.cc:325] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.944258 25727 raft_consensus.cc:351] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:49.944443 25727 raft_consensus.cc:628] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:49.944869 25727 consensus_queue.cc:227] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.947113 25727 ts_tablet_manager.cc:1073] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
W0619 00:11:49.947407 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.947886 25601 ts_tablet_manager.cc:1173] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:49.950582 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:49.953464 25602 ts_tablet_manager.cc:1173] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.929757 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet a85577b0beba4276b963e5c360544bc7
I0619 00:11:49.956719 25067 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 64 ms (attempt = 2)
I0619 00:11:49.956856 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.957164 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.957259 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.957567 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.957628 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.935950 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) successfully deleted
I0619 00:11:49.960057 25467 tablet_service.cc:759] Processing CreateTablet for tablet 3722b203a516466489c74ef6013eacae (table=default.table_6eabaa2225f44436880ea396f6bc6acd [id=a17737f9d2924cc09f372527fb91188b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.960603 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.960719 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3722b203a516466489c74ef6013eacae. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.960698 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:49.960857 25469 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.961145 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 81 ms (attempt = 2)
I0619 00:11:49.961302 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:49.961385 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:49.960579 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:49.961724 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 61 ms (attempt = 2)
I0619 00:11:49.961827 25736 tablet_replica.cc:261] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
W0619 00:11:49.961946 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.962054 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.962240 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.962419 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.962424 25082 master_service.cc:523] ReplaceTablet: received request to replace tablet f372b9fb174844eca1662c62039a3c71 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:49.962632 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(fc22ad98ed2e4e3eb2c62360626a6f87)
W0619 00:11:49.962654 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 because the tablet was not found. No further retry: Not found: Tablet not found: 985e23a32fa84104a06dd19d97117ab2
I0619 00:11:49.962914 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.963140 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.963510 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:49.963707 25462 tablet_service.cc:759] Processing CreateTablet for tablet 397ae10bb6d841b4ba561ec431d29864 (table=default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:49.963801 25736 raft_consensus.cc:2042] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.964138 25736 raft_consensus.cc:2069] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.965054 25330 ts_tablet_manager.cc:1173] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:49.947429 25727 ts_tablet_manager.cc:1028] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:49.965494 25727 tablet_bootstrap.cc:427] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:49.965770 25736 ts_tablet_manager.cc:1369] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:49.966179 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:49.966447 25067 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 279 ms (attempt = 5)
I0619 00:11:49.967561 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:49.968108 25727 tablet_bootstrap.cc:578] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
W0619 00:11:49.969498 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 1d10fb4561924e2e8d53c7a0b22a426e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 1d10fb4561924e2e8d53c7a0b22a426e already in progress: creating tablet
I0619 00:11:49.969760 25071 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 56 ms (attempt = 1)
I0619 00:11:49.964272 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 397ae10bb6d841b4ba561ec431d29864. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:49.969692 25080 catalog_manager.cc:3663] Got report from deleted tablet ac129ed44f68432f800420175aa0a630 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (replaced by tablet 35acf55f35064c1491122a13ff9f5744): Sending delete request for this tablet
I0619 00:11:49.970530 25080 catalog_manager.cc:3663] Got report from deleted tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
W0619 00:11:49.970962 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:49.971195 25071 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
I0619 00:11:49.971334 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
W0619 00:11:49.971567 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:49.971719 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.971943 25602 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:49.972331 25330 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.972759 25741 tablet_replica.cc:261] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:49.972980 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.973196 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.973374 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.973556 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.973779 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.973623 25730 ts_tablet_manager.cc:1382] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:49.973987 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.974164 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(baf80fd968c44e679987eb4ebe4473de)
I0619 00:11:49.974277 25730 log.cc:986] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/1d10fb4561924e2e8d53c7a0b22a426e
I0619 00:11:49.974350 25741 raft_consensus.cc:2042] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:49.974591 25741 raft_consensus.cc:2069] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:49.975136 25730 ts_tablet_manager.cc:1403] T 1d10fb4561924e2e8d53c7a0b22a426e P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:49.976258 25741 ts_tablet_manager.cc:1369] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:49.982798 25087 catalog_manager.cc:3663] Got report from deleted tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:49.983417 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.984655 25330 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.985584 25330 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.986346 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.987684 25330 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:49.988729 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:49.989012 25071 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 63 ms (attempt = 1)
I0619 00:11:49.991696 25756 raft_consensus.cc:457] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:49.992024 25756 raft_consensus.cc:2732] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.807s (starting election)
I0619 00:11:49.992511 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 541 ms (attempt = 6)
W0619 00:11:49.993415 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:49.992347 25756 raft_consensus.cc:479] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:49.994504 25756 leader_election.cc:243] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:49.995297 25756 leader_election.cc:243] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:49.999567 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3d4be1da5dd54e91b7a565ceded2dae3" }
I0619 00:11:50.001015 25068 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 1073 ms (attempt = 7)
W0619 00:11:50.001302 25068 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.001636 25756 raft_consensus.cc:457] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.001881 25756 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.071s (starting election)
I0619 00:11:50.002225 25724 tablet_bootstrap.cc:427] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.002111 25756 raft_consensus.cc:479] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.002564 25724 ts_tablet_manager.cc:1045] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.097s	user 0.002s	sys 0.011s
I0619 00:11:50.003049 25756 leader_election.cc:243] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.003410 25756 leader_election.cc:243] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.004024 25724 raft_consensus.cc:325] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.004689 25724 raft_consensus.cc:351] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.004901 25724 raft_consensus.cc:628] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:50.005460 25724 consensus_queue.cc:227] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.006132 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 163 ms (attempt = 4)
W0619 00:11:50.006386 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.006749 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 278 ms (attempt = 5)
W0619 00:11:50.006999 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_31a3ade4f34e4e7a96d0bc91603e471d [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
W0619 00:11:50.007469 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:50.007709 25071 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 72 ms (attempt = 2)
I0619 00:11:50.006305 25741 ts_tablet_manager.cc:1382] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.008746 25741 log.cc:986] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/baf80fd968c44e679987eb4ebe4473de
I0619 00:11:50.009260 25741 ts_tablet_manager.cc:1403] T baf80fd968c44e679987eb4ebe4473de P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:50.009325 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.012099 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 397ae10bb6d841b4ba561ec431d29864
W0619 00:11:50.012392 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.012382 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.012667 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet f372b9fb174844eca1662c62039a3c71
I0619 00:11:50.012785 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.012958 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.013171 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.013319 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.013701 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.013794 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.014019 25082 catalog_manager.cc:4596] ReplaceTablet: tablet f372b9fb174844eca1662c62039a3c71 deleted and replaced by tablet 30a9d9fda8d34caeb8242c372439321b
I0619 00:11:50.015833 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.015498 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_da957178ddc241ed8e414551ec468a12"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:50.016474 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da957178ddc241ed8e414551ec468a12 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.015491 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:50.020598 25330 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.020725 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.021486 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.023725 25469 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.023847 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.023893 25467 ts_tablet_manager.cc:1173] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.024811 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.027596 25083 master_service.cc:523] ReplaceTablet: received request to replace tablet 397ae10bb6d841b4ba561ec431d29864 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:50.029088 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 82 ms (attempt = 3)
W0619 00:11:50.031085 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.031795 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.033967 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.038631 25469 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.038946 25467 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.039275 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 555 ms (attempt = 6)
I0619 00:11:50.039361 25464 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:50.039543 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.040463 25325 tablet_service.cc:759] Processing CreateTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e (table=default.table_0a0cd847ea574c2eb82ae9dfaf67e95f [id=040c5029d5054cc48c9f066058cb6a45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.042052 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 075076aaa16e4ea8a596a7a8c52f103e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.046231 25736 ts_tablet_manager.cc:1382] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.046505 25736 log.cc:986] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/fc22ad98ed2e4e3eb2c62360626a6f87
I0619 00:11:50.046499 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet baf80fd968c44e679987eb4ebe4473de on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.046805 25724 ts_tablet_manager.cc:1073] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.044s	user 0.005s	sys 0.000s
I0619 00:11:50.048003 25330 tablet_service.cc:800] Processing DeleteTablet for tablet baf80fd968c44e679987eb4ebe4473de with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.049437 25736 ts_tablet_manager.cc:1403] T fc22ad98ed2e4e3eb2c62360626a6f87 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:50.052863 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 69 ms (attempt = 3)
W0619 00:11:50.053165 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:50.053697 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) successfully deleted
I0619 00:11:50.053966 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_8687a04df04444c6a3f710f1b9884ce5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:50.054878 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8687a04df04444c6a3f710f1b9884ce5 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.054957 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_62e2a91242c24687b8c59b83aa396c02"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:50.055840 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_62e2a91242c24687b8c59b83aa396c02 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.057935 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.058897 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.059550 25470 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.059972 25741 tablet_replica.cc:261] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:50.060247 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.060324 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.060468 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.060988 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.061277 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.061305 25470 tablet_service.cc:759] Processing CreateTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e (table=default.table_0a0cd847ea574c2eb82ae9dfaf67e95f [id=040c5029d5054cc48c9f066058cb6a45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.061511 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.061775 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.061982 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.062181 25741 raft_consensus.cc:2042] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.062458 25741 raft_consensus.cc:2069] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:50.066390 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 075076aaa16e4ea8a596a7a8c52f103e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.066836 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.051287 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.051517 25601 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.058274 25604 tablet_service.cc:759] Processing CreateTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e (table=default.table_0a0cd847ea574c2eb82ae9dfaf67e95f [id=040c5029d5054cc48c9f066058cb6a45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.068002 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 075076aaa16e4ea8a596a7a8c52f103e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.068970 25776 raft_consensus.cc:457] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.069228 25776 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.688s (starting election)
I0619 00:11:50.069485 25776 raft_consensus.cc:479] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.070544 25776 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.070973 25776 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.071856 25725 tablet_bootstrap.cc:427] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:50.072175 25725 ts_tablet_manager.cc:1045] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.173s	user 0.006s	sys 0.010s
I0619 00:11:50.075593 25736 tablet_replica.cc:261] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:50.058337 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet baf80fd968c44e679987eb4ebe4473de (table default.table_5e925f129a1043b7ab80c112a0ee33e0 [id=eeb1072dfe27492990409ab910065b9c]) successfully deleted
W0619 00:11:50.076133 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.076372 25067 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 55 ms (attempt = 1)
W0619 00:11:50.076769 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.076977 25067 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 48 ms (attempt = 2)
W0619 00:11:50.077339 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:50.077548 25067 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 89 ms (attempt = 3)
W0619 00:11:50.077958 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet ac129ed44f68432f800420175aa0a630 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ac129ed44f68432f800420175aa0a630 already in progress: creating tablet
I0619 00:11:50.078163 25067 catalog_manager.cc:2891] Scheduling retry of ac129ed44f68432f800420175aa0a630 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 100 ms (attempt = 3)
W0619 00:11:50.076133 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:50.078929 25724 ts_tablet_manager.cc:1028] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.079265 25724 tablet_bootstrap.cc:427] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:50.079325 25071 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 19 ms (attempt = 1)
I0619 00:11:50.080412 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.081965 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]) successfully deleted
W0619 00:11:50.083197 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.084226 25071 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 26 ms (attempt = 1)
I0619 00:11:50.086391 25724 tablet_bootstrap.cc:578] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:50.086696 25756 raft_consensus.cc:457] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.085942 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:50.087383 25470 ts_tablet_manager.cc:1173] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.088937 25725 raft_consensus.cc:325] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.089586 25725 raft_consensus.cc:351] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.089838 25725 raft_consensus.cc:628] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:50.090301 25725 consensus_queue.cc:227] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.093835 25741 ts_tablet_manager.cc:1369] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.086974 25756 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.160s (starting election)
I0619 00:11:50.089685 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(ac129ed44f68432f800420175aa0a630)
W0619 00:11:50.102766 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.090343 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:50.092887 25725 ts_tablet_manager.cc:1073] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.021s	user 0.003s	sys 0.001s
I0619 00:11:50.103479 25071 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 34 ms (attempt = 2)
I0619 00:11:50.103611 25725 ts_tablet_manager.cc:1028] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
W0619 00:11:50.088734 25558 leader_election.cc:306] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.104015 25725 tablet_bootstrap.cc:427] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:50.105273 25624 raft_consensus.cc:1081] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:50.107081 25725 tablet_bootstrap.cc:578] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:50.107378 25743 consensus_queue.cc:963] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:50.107899 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.095003 25756 raft_consensus.cc:479] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.111166 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:50.112565 25553 leader_election.cc:306] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.112644 25756 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.112843 25553 leader_election.cc:270] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:50.113061 25756 leader_election.cc:243] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.113845 25776 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.778s (election complete)
I0619 00:11:50.113837 25354 raft_consensus.cc:2784] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:50.114068 25776 raft_consensus.cc:2487] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:50.114662 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.117069 25470 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.117805 25741 ts_tablet_manager.cc:1382] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.118041 25741 log.cc:986] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/1d10fb4561924e2e8d53c7a0b22a426e
I0619 00:11:50.118505 25741 ts_tablet_manager.cc:1403] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:50.119699 25354 raft_consensus.cc:1081] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W0619 00:11:50.120182 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:50.121222 25071 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 52 ms (attempt = 2)
I0619 00:11:50.126215 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.126214 25727 tablet_bootstrap.cc:427] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:50.126561 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:50.126790 25727 ts_tablet_manager.cc:1045] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.161s	user 0.004s	sys 0.012s
I0619 00:11:50.127216 25330 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.127308 25080 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 23 ms (attempt = 1)
W0619 00:11:50.127607 25080 catalog_manager.cc:3945] Failed to send fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.128106 25727 raft_consensus.cc:325] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.128743 25727 raft_consensus.cc:351] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.128918 25727 raft_consensus.cc:628] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:50.129459 25727 consensus_queue.cc:227] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.131155 25727 ts_tablet_manager.cc:1073] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:11:50.132576 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.142405 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.146147 25462 ts_tablet_manager.cc:1173] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.146423 25743 consensus_queue.cc:963] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:50.149734 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:50.149788 25325 ts_tablet_manager.cc:1173] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.150327 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.150897 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 165 ms (attempt = 4)
I0619 00:11:50.151039 25725 tablet_bootstrap.cc:427] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
W0619 00:11:50.151187 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.152140 25067 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 65 ms (attempt = 2)
W0619 00:11:50.152416 25067 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.152369 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_dda28a7d27fb424e8e0e9f6657a52c6c" } new_table_name: "default.table_48a6321c18954a77805c579d820e8d00" alter_external_catalogs: true
I0619 00:11:50.152498 25725 ts_tablet_manager.cc:1045] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.049s	user 0.006s	sys 0.001s
I0619 00:11:50.153337 25604 ts_tablet_manager.cc:1173] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.154157 25330 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.154413 25603 ts_tablet_manager.cc:1173] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:50.154525 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 60dfd91c98d94bfbbc6416268c65c378 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 60dfd91c98d94bfbbc6416268c65c378 already in progress: creating tablet
I0619 00:11:50.154794 25071 catalog_manager.cc:2891] Scheduling retry of 60dfd91c98d94bfbbc6416268c65c378 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 72 ms (attempt = 2)
W0619 00:11:50.155122 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.155330 25071 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 37 ms (attempt = 1)
W0619 00:11:50.155668 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times.
W0619 00:11:50.155076 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.156510 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 154 ms (attempt = 4)
I0619 00:11:50.156308 25727 ts_tablet_manager.cc:1028] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:50.156946 25727 tablet_bootstrap.cc:427] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:50.157107 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fb8e94875a8347a0a473c95a77c38795" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:50.157654 25622 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.831s (vote granted)
I0619 00:11:50.158299 25622 raft_consensus.cc:2245] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:50.159029 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:50.159315 25325 tablet_service.cc:759] Processing CreateTablet for tablet 3537c79c30a1401e8a78b387f08eff58 (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.159281 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:50.160022 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:50.160305 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:50.160557 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:50.160846 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:50.161070 25736 raft_consensus.cc:2042] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.161361 25736 raft_consensus.cc:2069] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:50.162412 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.163684 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1d10fb4561924e2e8d53c7a0b22a426e" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:50.164489 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fb8e94875a8347a0a473c95a77c38795" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:50.165020 25489 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.242s (vote granted)
I0619 00:11:50.165109 25736 ts_tablet_manager.cc:1369] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.165299 25489 raft_consensus.cc:2245] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:50.171239 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e3708e05f6054306b5bf7f1df2140e3d" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:50.171833 25489 raft_consensus.cc:2206] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:50.170255 25725 raft_consensus.cc:325] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:50.173107 25290 leader_election.cc:306] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 1d10fb4561924e2e8d53c7a0b22a426e
I0619 00:11:50.173363 25724 tablet_bootstrap.cc:427] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.173738 25724 ts_tablet_manager.cc:1045] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.095s	user 0.005s	sys 0.002s
I0619 00:11:50.173373 25725 raft_consensus.cc:351] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:50.174862 25290 leader_election.cc:306] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.175114 25290 leader_election.cc:270] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:50.175580 25622 raft_consensus.cc:2220] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:11:50.134296 25080 catalog_manager.cc:3663] Got report from deleted tablet dc04126054714196af6bdf7e771ffab0 (table default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:50.174839 25725 raft_consensus.cc:628] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:50.162897 25083 catalog_manager.cc:2891] Scheduling retry of aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC with a delay of 63 ms (attempt = 1)
I0619 00:11:50.167268 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:50.175968 25083 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aaf62da07a9740c1aa44359d3f604513 consensus state has no leader
I0619 00:11:50.176502 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) successfully deleted
I0619 00:11:50.176780 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dc04126054714196af6bdf7e771ffab0 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
W0619 00:11:50.176986 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 1d10fb4561924e2e8d53c7a0b22a426e because the tablet was not found. No further retry: Not found: Tablet not found: 1d10fb4561924e2e8d53c7a0b22a426e
W0619 00:11:50.178059 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.178278 25067 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
I0619 00:11:50.178714 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
W0619 00:11:50.179412 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:50.177284 25725 consensus_queue.cc:227] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.179626 25067 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 44 ms (attempt = 1)
W0619 00:11:50.181381 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:50.181638 25067 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 92 ms (attempt = 3)
W0619 00:11:50.182086 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:50.182330 25067 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 83 ms (attempt = 3)
W0619 00:11:50.182816 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet baf80fd968c44e679987eb4ebe4473de because the tablet was not found. No further retry: Not found: Tablet not found: baf80fd968c44e679987eb4ebe4473de
I0619 00:11:50.182843 25724 raft_consensus.cc:325] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:50.183277 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.183495 25724 raft_consensus.cc:351] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.183630 25067 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 84 ms (attempt = 3)
I0619 00:11:50.183732 25724 raft_consensus.cc:628] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:50.184132 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.184373 25067 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 59 ms (attempt = 2)
I0619 00:11:50.184859 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3537c79c30a1401e8a78b387f08eff58. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:50.187127 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.188011 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 3 got 0
I0619 00:11:50.189571 25083 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]). Expected version 1 got 0
I0619 00:11:50.190114 25727 tablet_bootstrap.cc:578] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:50.190287 25083 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 64 ms (attempt = 1)
W0619 00:11:50.190403 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.190541 25083 catalog_manager.cc:3945] Failed to send 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
W0619 00:11:50.181687 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:50.196241 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_31a3ade4f34e4e7a96d0bc91603e471d" } new_table_name: "default.table_00f27d6b83d24c4e8088e0b0ac0d0cca" alter_external_catalogs: true
W0619 00:11:50.198319 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.198037 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.198778 25071 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 29 ms (attempt = 1)
W0619 00:11:50.199121 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:50.200117 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:50.200595 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 277 ms (attempt = 5)
I0619 00:11:50.184216 25724 consensus_queue.cc:227] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.201316 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 3 got 0
I0619 00:11:50.202044 25724 ts_tablet_manager.cc:1073] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.000s
I0619 00:11:50.202404 25724 ts_tablet_manager.cc:1028] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.196570 25797 tablet.cc:1200] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3
W0619 00:11:50.203012 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.207209 25083 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 26 ms (attempt = 1)
W0619 00:11:50.207484 25083 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:50.208007 25724 tablet_bootstrap.cc:427] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:50.200902 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.209344 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:50.212317 25725 ts_tablet_manager.cc:1073] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.059s	user 0.004s	sys 0.000s
I0619 00:11:50.213064 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:50.213744 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.213886 25725 ts_tablet_manager.cc:1028] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:50.214360 25725 tablet_bootstrap.cc:427] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:50.215847 25462 tablet_service.cc:759] Processing CreateTablet for tablet 3537c79c30a1401e8a78b387f08eff58 (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.216445 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3537c79c30a1401e8a78b387f08eff58. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.217208 25470 tablet_service.cc:759] Processing CreateTablet for tablet 30a9d9fda8d34caeb8242c372439321b (table=default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.217309 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_8687a04df04444c6a3f710f1b9884ce5" }
I0619 00:11:50.217810 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 30a9d9fda8d34caeb8242c372439321b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.218245 25067 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 80 ms (attempt = 3)
W0619 00:11:50.218529 25067 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.222790 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.225241 25462 ts_tablet_manager.cc:1173] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.225790 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.226359 25470 ts_tablet_manager.cc:1173] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.227157 25725 tablet_bootstrap.cc:578] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:50.229692 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.230356 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 33 ms (attempt = 1)
I0619 00:11:50.232163 25724 tablet_bootstrap.cc:578] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:50.233788 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.234186 25604 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.235908 25604 tablet_service.cc:759] Processing CreateTablet for tablet 3537c79c30a1401e8a78b387f08eff58 (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.236481 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3537c79c30a1401e8a78b387f08eff58. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.240633 25068 catalog_manager.cc:2891] Scheduling retry of aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC with a delay of 55 ms (attempt = 2)
W0619 00:11:50.240900 25068 catalog_manager.cc:2910] Async tablet task aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet aaf62da07a9740c1aa44359d3f604513 consensus state has no leader
I0619 00:11:50.243973 25082 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 3 got 0
I0619 00:11:50.244277 25083 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]). Expected version 1 got 0
I0619 00:11:50.244976 25083 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 48 ms (attempt = 1)
W0619 00:11:50.245218 25083 catalog_manager.cc:3945] Failed to send 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.246302 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.246775 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 1035 ms (attempt = 7)
W0619 00:11:50.247014 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.247285 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.248427 25083 catalog_manager.cc:3663] Got report from deleted tablet 3b80a302aacb418091aee54ab4be7249 (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet acc89522b693498c987ca5d8ef8275ea): Sending delete request for this tablet
I0619 00:11:50.249572 25603 tablet_service.cc:759] Processing CreateTablet for tablet 30a9d9fda8d34caeb8242c372439321b (table=default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.250186 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 30a9d9fda8d34caeb8242c372439321b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.252240 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 60dfd91c98d94bfbbc6416268c65c378 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.252912 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.253304 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:50.257717 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 67 ms (attempt = 2)
W0619 00:11:50.257776 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.257977 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.259824 25604 ts_tablet_manager.cc:1173] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.261761 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:50.262058 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.262431 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.262828 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.263496 25724 tablet_bootstrap.cc:427] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.264130 25724 ts_tablet_manager.cc:1045] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.057s	user 0.003s	sys 0.003s
I0619 00:11:50.267355 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_4b9f3e0c22b54ee381137efdf5107650"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:50.267772 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
W0619 00:11:50.268782 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4b9f3e0c22b54ee381137efdf5107650 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.269109 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.271819 25083 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4] alter complete (version 3)
I0619 00:11:50.273315 25797 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 3 got alter request for version 3
I0619 00:11:50.273895 25736 ts_tablet_manager.cc:1382] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.274410 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:50.274286 25736 log.cc:986] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/ac129ed44f68432f800420175aa0a630
I0619 00:11:50.275554 25736 ts_tablet_manager.cc:1403] T ac129ed44f68432f800420175aa0a630 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
W0619 00:11:50.276170 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.277362 25256 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:50.277899 25256 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.278501 25256 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.279089 25256 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.266232 25724 raft_consensus.cc:325] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.286039 25603 ts_tablet_manager.cc:1173] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.286965 25799 tablet.cc:1200] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3
I0619 00:11:50.289476 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 35 ms (attempt = 2)
I0619 00:11:50.289892 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.290366 25604 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.290570 25730 tablet_replica.cc:261] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:50.290870 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:50.289937 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_0a2ef368d0f945eaaeeb3e58d57ef718" }
I0619 00:11:50.291076 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:50.291260 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:50.291440 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:50.291641 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:50.291851 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:50.292022 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:50.292186 25730 raft_consensus.cc:2042] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.292428 25730 raft_consensus.cc:2069] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:50.290314 25602 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.293860 25730 ts_tablet_manager.cc:1369] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.294638 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 45 ms (attempt = 2)
W0619 00:11:50.295193 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
W0619 00:11:50.290978 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:50.299352 25067 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 136 ms (attempt = 4)
I0619 00:11:50.299479 25068 catalog_manager.cc:2891] Scheduling retry of aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC with a delay of 76 ms (attempt = 3)
I0619 00:11:50.299720 25325 ts_tablet_manager.cc:1173] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:50.299880 25068 catalog_manager.cc:2910] Async tablet task aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet aaf62da07a9740c1aa44359d3f604513 consensus state has no leader
W0619 00:11:50.300060 25067 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.302182 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.302876 25776 raft_consensus.cc:457] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.303110 25776 raft_consensus.cc:2732] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.895s (starting election)
I0619 00:11:50.303349 25776 raft_consensus.cc:479] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.304347 25776 leader_election.cc:243] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.304754 25776 leader_election.cc:243] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.306406 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 03522c48c50943b38cae5aa88562e583
I0619 00:11:50.306742 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.307576 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 530 ms (attempt = 6)
W0619 00:11:50.302402 25290 raft_consensus.cc:2442] T 1d10fb4561924e2e8d53c7a0b22a426e P c007a00e65ad41a79da1a0c1050c55b4: Unable to run election callback: Service unavailable: Thread pool token was shut down
W0619 00:11:50.308781 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:50.307963 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.309427 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.313527 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "53688f0e4a604bc4a07dec9278a8dad6" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:50.313580 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_dd9523cf3fda45cd8bc544abb9dfba01"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:50.314644 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dd9523cf3fda45cd8bc544abb9dfba01 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.316721 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:50.318387 25553 leader_election.cc:306] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.320375 25082 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 3 got 0
I0619 00:11:50.320472 25725 tablet_bootstrap.cc:427] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:50.320964 25725 ts_tablet_manager.cc:1045] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.107s	user 0.005s	sys 0.005s
W0619 00:11:50.322047 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 147186us
W0619 00:11:50.324509 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:50.324995 25290 leader_election.cc:379] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.325224 25290 leader_election.cc:270] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:50.325271 25071 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 77 ms (attempt = 2)
I0619 00:11:50.325681 25756 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.212s (election complete)
I0619 00:11:50.325918 25756 raft_consensus.cc:2539] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:50.326126 25756 raft_consensus.cc:457] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:50.326344 25756 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.049s (starting election)
I0619 00:11:50.326545 25756 raft_consensus.cc:2784] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:50.326880 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 112 ms (attempt = 3)
W0619 00:11:50.327143 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.327960 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.328089 25290 leader_election.cc:379] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.330518 25325 tablet_service.cc:759] Processing CreateTablet for tablet 30a9d9fda8d34caeb8242c372439321b (table=default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.331125 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 30a9d9fda8d34caeb8242c372439321b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.331261 25071 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 266 ms (attempt = 5)
W0619 00:11:50.331718 25071 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.332450 25290 leader_election.cc:393] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Service unavailable: T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:11:50.332917 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_2e052476ba2c4ab08ef5774928db6084"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:50.333808 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:50.334151 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2e052476ba2c4ab08ef5774928db6084 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.335422 25756 raft_consensus.cc:479] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.336354 25756 leader_election.cc:243] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.336733 25756 leader_election.cc:243] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.332587 25725 raft_consensus.cc:325] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.340497 25725 raft_consensus.cc:351] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:50.341106 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.341325 25330 tablet_service.cc:800] Processing DeleteTablet for tablet dc04126054714196af6bdf7e771ffab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.341553 25071 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 58 ms (attempt = 2)
I0619 00:11:50.341791 25741 tablet_replica.cc:261] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:50.342037 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:50.342116 25290 leader_election.cc:393] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:50.342180 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 82 ms (attempt = 3)
I0619 00:11:50.342388 25290 leader_election.cc:270] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:50.342443 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
W0619 00:11:50.342819 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:50.343073 25067 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 137 ms (attempt = 4)
I0619 00:11:50.343466 25330 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.343493 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet 3b80a302aacb418091aee54ab4be7249 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:50.343854 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:50.344055 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:50.344246 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:50.344054 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fb8e94875a8347a0a473c95a77c38795" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:50.344457 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:50.344646 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:50.344856 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(dc04126054714196af6bdf7e771ffab0)
I0619 00:11:50.344578 25602 tablet_service.cc:759] Processing CreateTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 (table=default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.345024 25741 raft_consensus.cc:2042] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.345046 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 1047 ms (attempt = 7)
I0619 00:11:50.345293 25741 raft_consensus.cc:2069] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:50.345324 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.345508 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a52cbcbe35374d518bca1e0fc24d88d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.345713 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.344527 25624 raft_consensus.cc:2784] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:50.345836 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fb8e94875a8347a0a473c95a77c38795" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:50.346211 25624 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.158s (vote granted)
I0619 00:11:50.346277 25489 raft_consensus.cc:2784] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:50.346518 25489 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.971s (vote granted)
I0619 00:11:50.347234 25601 tablet_service.cc:759] Processing CreateTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 (table=default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.347805 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7f60ea5c5def47ee846d0186bbc90c18. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.348749 25323 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.348842 25326 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.349503 25330 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.344820 25604 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.350700 25741 ts_tablet_manager.cc:1369] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.350816 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.351486 25624 raft_consensus.cc:2245] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:50.353391 25727 tablet_bootstrap.cc:427] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:50.353727 25727 ts_tablet_manager.cc:1045] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.197s	user 0.006s	sys 0.000s
I0619 00:11:50.355480 25727 raft_consensus.cc:325] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.355957 25797 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 3 got alter request for version 3
I0619 00:11:50.356091 25727 raft_consensus.cc:351] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.340798 25725 raft_consensus.cc:628] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:50.356807 25603 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.356262 25727 raft_consensus.cc:628] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:50.357251 25727 consensus_queue.cc:227] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.357558 25725 consensus_queue.cc:227] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.361150 25725 ts_tablet_manager.cc:1073] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.000s
I0619 00:11:50.363721 25489 raft_consensus.cc:2245] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:50.366727 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "53688f0e4a604bc4a07dec9278a8dad6" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:50.370807 25080 catalog_manager.cc:3663] Got report from deleted tablet 60dfd91c98d94bfbbc6416268c65c378 (table default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:50.371708 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 60dfd91c98d94bfbbc6416268c65c378 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.374384 25756 raft_consensus.cc:2732] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.110s (election complete)
I0619 00:11:50.374476 25330 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.374719 25756 raft_consensus.cc:2487] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:50.375039 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:50.375624 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.377486 25083 catalog_manager.cc:3663] Got report from deleted tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:50.378243 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:50.378562 25068 catalog_manager.cc:2891] Scheduling retry of aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC with a delay of 131 ms (attempt = 4)
W0619 00:11:50.378882 25068 catalog_manager.cc:2910] Async tablet task aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet aaf62da07a9740c1aa44359d3f604513 consensus state has no leader
I0619 00:11:50.378983 25776 raft_consensus.cc:457] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.379240 25776 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.532s (starting election)
I0619 00:11:50.379489 25776 raft_consensus.cc:479] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.380467 25776 leader_election.cc:243] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.380883 25776 leader_election.cc:243] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.382073 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aaf62da07a9740c1aa44359d3f604513" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:50.384187 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 293 ms (attempt = 5)
W0619 00:11:50.384212 25553 leader_election.cc:306] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:50.384822 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:50.385182 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 68 ms (attempt = 3)
I0619 00:11:50.361776 25725 ts_tablet_manager.cc:1028] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:50.386091 25727 ts_tablet_manager.cc:1073] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.002s
I0619 00:11:50.386667 25725 tablet_bootstrap.cc:427] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:50.387297 25727 ts_tablet_manager.cc:1028] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:50.387663 25727 tablet_bootstrap.cc:427] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:50.387708 25083 catalog_manager.cc:3663] Got report from deleted tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:50.388334 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 985e23a32fa84104a06dd19d97117ab2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:50.391232 25727 tablet_bootstrap.cc:578] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
W0619 00:11:50.392256 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.391819 25799 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Already running schema version 3 got alter request for version 3
W0619 00:11:50.396324 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.396870 25741 ts_tablet_manager.cc:1382] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.397114 25741 log.cc:986] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/dc04126054714196af6bdf7e771ffab0
I0619 00:11:50.397593 25741 ts_tablet_manager.cc:1403] T dc04126054714196af6bdf7e771ffab0 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:50.398607 25725 tablet_bootstrap.cc:578] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:50.398699 25558 leader_election.cc:306] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.399039 25558 leader_election.cc:270] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:50.399488 25776 raft_consensus.cc:2732] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.742s (election complete)
I0619 00:11:50.399734 25776 raft_consensus.cc:2487] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:50.402724 25601 ts_tablet_manager.cc:1173] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.402806 25290 leader_election.cc:379] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.398747 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:50.399380 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.400969 25797 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 3 got alter request for version 3
I0619 00:11:50.401568 25462 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.401851 25463 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.403239 25290 leader_election.cc:270] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:50.403406 25736 tablet_replica.cc:261] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:50.403664 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:50.403868 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:50.403808 25804 raft_consensus.cc:2732] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.739s (election complete)
I0619 00:11:50.404053 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:50.396270 25724 raft_consensus.cc:351] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.404136 25804 raft_consensus.cc:2539] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:50.404244 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:50.404285 25724 raft_consensus.cc:628] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:50.404469 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:50.404690 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:50.404896 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(60dfd91c98d94bfbbc6416268c65c378)
I0619 00:11:50.405081 25736 raft_consensus.cc:2042] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.405195 25602 ts_tablet_manager.cc:1173] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.405356 25736 raft_consensus.cc:2069] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:50.404940 25724 consensus_queue.cc:227] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.408095 25724 ts_tablet_manager.cc:1073] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.143s	user 0.003s	sys 0.001s
I0619 00:11:50.408471 25724 ts_tablet_manager.cc:1028] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.408838 25724 tablet_bootstrap.cc:427] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:50.409595 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.403960 25290 leader_election.cc:379] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.415359 25330 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.402372 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aaf62da07a9740c1aa44359d3f604513" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:50.415834 25736 ts_tablet_manager.cc:1369] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:50.419287 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.404989 25730 ts_tablet_manager.cc:1382] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.420212 25730 log.cc:986] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/3b80a302aacb418091aee54ab4be7249
I0619 00:11:50.420189 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.420984 25730 ts_tablet_manager.cc:1403] T 3b80a302aacb418091aee54ab4be7249 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:50.421059 25330 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.421766 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet dc04126054714196af6bdf7e771ffab0 (table default.table_706cbf40bb854d3ebbdbdbf69f4e15d6 [id=091a66f5716e463293291bf5aba8723a]) successfully deleted
W0619 00:11:50.422232 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet ac129ed44f68432f800420175aa0a630 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ac129ed44f68432f800420175aa0a630 already in progress: creating tablet
I0619 00:11:50.422446 25067 catalog_manager.cc:2891] Scheduling retry of ac129ed44f68432f800420175aa0a630 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 151 ms (attempt = 4)
W0619 00:11:50.422842 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:50.423050 25067 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 39 ms (attempt = 2)
W0619 00:11:50.423413 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc22ad98ed2e4e3eb2c62360626a6f87 already in progress: creating tablet
I0619 00:11:50.423633 25067 catalog_manager.cc:2891] Scheduling retry of fc22ad98ed2e4e3eb2c62360626a6f87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 543 ms (attempt = 6)
I0619 00:11:50.423701 25804 raft_consensus.cc:593] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
W0619 00:11:50.424037 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.425482 25468 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:50.426703 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.426775 25805 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Already running schema version 3 got alter request for version 3
I0619 00:11:50.424249 25067 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 87 ms (attempt = 3)
I0619 00:11:50.427989 25805 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Already running schema version 3 got alter request for version 3
I0619 00:11:50.428373 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 157 ms (attempt = 4)
W0619 00:11:50.428679 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
W0619 00:11:50.429113 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.429291 25724 tablet_bootstrap.cc:578] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:50.429343 25067 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 78 ms (attempt = 2)
I0619 00:11:50.430971 25069 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 1068 ms (attempt = 7)
W0619 00:11:50.431237 25069 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
W0619 00:11:50.431741 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.433682 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.434546 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:50.435338 25468 tablet_service.cc:759] Processing CreateTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 (table=default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.435848 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.435961 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a52cbcbe35374d518bca1e0fc24d88d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.436265 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.437094 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_33483e8524674ab0a5207eb14818073b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:50.437691 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet ac129ed44f68432f800420175aa0a630 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) successfully deleted
W0619 00:11:50.438004 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_33483e8524674ab0a5207eb14818073b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:50.438872 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet ac129ed44f68432f800420175aa0a630 because the tablet was not found. No further retry: Not found: Tablet not found: ac129ed44f68432f800420175aa0a630
W0619 00:11:50.439640 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 because the tablet was not found. No further retry: Not found: Tablet not found: fc22ad98ed2e4e3eb2c62360626a6f87
I0619 00:11:50.439985 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 3b80a302aacb418091aee54ab4be7249 (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) successfully deleted
W0619 00:11:50.440405 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:50.440603 25071 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
W0619 00:11:50.440917 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.442003 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet c18355df78d344839a56189c7549ab1f from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:50.442886 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.443315 25083 catalog_manager.cc:3663] Got report from deleted tablet 292b164659e343158de8e7e3eeb9be31 (table default.table_aecb1278dc98420aa83aaae47fb56937 [id=e2a7900f428948bab0aefd132d8874d4]) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0): Sending delete request for this tablet
I0619 00:11:50.443588 25601 tablet_service.cc:759] Processing CreateTablet for tablet 7c57c5ca55f24621b66197a3698504bd (table=default.table_4b9f3e0c22b54ee381137efdf5107650 [id=68b201dd31454f02838aaa70b04c3c31]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.444203 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7c57c5ca55f24621b66197a3698504bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.444169 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.447791 25602 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.449852 25067 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC with a delay of 286 ms (attempt = 5)
I0619 00:11:50.450065 25725 tablet_bootstrap.cc:427] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
W0619 00:11:50.450232 25067 catalog_manager.cc:2910] Async tablet task fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fb8e94875a8347a0a473c95a77c38795 consensus state has no leader
I0619 00:11:50.450511 25725 ts_tablet_manager.cc:1045] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.064s	user 0.005s	sys 0.006s
I0619 00:11:50.450695 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 140 ms (attempt = 4)
W0619 00:11:50.450953 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_95a48b4d8a824265bb7291a68eb33dc3 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.452015 25725 raft_consensus.cc:325] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.452661 25725 raft_consensus.cc:351] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.457741 25071 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 41 ms (attempt = 2)
W0619 00:11:50.458510 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.459192 25725 raft_consensus.cc:628] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:50.459498 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 21 ms (attempt = 1)
I0619 00:11:50.459656 25725 consensus_queue.cc:227] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:50.460937 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
W0619 00:11:50.460228 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.461691 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 63 ms (attempt = 1)
I0619 00:11:50.461607 25462 tablet_service.cc:759] Processing CreateTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 (table=default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.462249 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7f60ea5c5def47ee846d0186bbc90c18. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.466763 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.468363 25082 catalog_manager.cc:3663] Got report from deleted tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
W0619 00:11:50.462057 25558 leader_election.cc:306] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.468685 25558 leader_election.cc:270] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:50.468811 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.469244 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.469869 25730 tablet_replica.cc:261] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:50.470160 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:50.470386 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:50.464092 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.464180 25326 tablet_service.cc:759] Processing CreateTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 (table=default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.470628 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:50.470903 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:50.463799 25330 tablet_service.cc:759] Processing CreateTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 (table=default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.471138 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:50.471338 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:50.471573 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:50.471801 25730 raft_consensus.cc:2042] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.472360 25468 ts_tablet_manager.cc:1173] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.472590 25725 ts_tablet_manager.cc:1073] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.000s
I0619 00:11:50.473007 25725 ts_tablet_manager.cc:1028] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:50.473386 25725 tablet_bootstrap.cc:427] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:50.474493 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.475397 25323 tablet_service.cc:759] Processing CreateTablet for tablet 7c57c5ca55f24621b66197a3698504bd (table=default.table_4b9f3e0c22b54ee381137efdf5107650 [id=68b201dd31454f02838aaa70b04c3c31]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.475662 25725 tablet_bootstrap.cc:578] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:50.476282 25796 raft_consensus.cc:457] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.476534 25796 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.753s (starting election)
I0619 00:11:50.476692 25333 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.476845 25796 raft_consensus.cc:479] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.477818 25796 leader_election.cc:243] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.478224 25796 leader_election.cc:243] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.476155 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 985e23a32fa84104a06dd19d97117ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.463279 25325 ts_tablet_manager.cc:1173] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.481632 25741 tablet_replica.cc:261] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:50.481886 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:50.482081 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:50.482260 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:50.482445 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:50.482681 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:50.482868 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:50.483047 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(985e23a32fa84104a06dd19d97117ab2)
I0619 00:11:50.483072 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.483211 25741 raft_consensus.cc:2042] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.483464 25741 raft_consensus.cc:2069] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:50.483489 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:50.484246 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7f60ea5c5def47ee846d0186bbc90c18. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:50.485317 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.486634 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:50.486770 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 60dfd91c98d94bfbbc6416268c65c378 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.488029 25067 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 132 ms (attempt = 4)
I0619 00:11:50.489126 25463 tablet_service.cc:759] Processing CreateTablet for tablet 7c57c5ca55f24621b66197a3698504bd (table=default.table_4b9f3e0c22b54ee381137efdf5107650 [id=68b201dd31454f02838aaa70b04c3c31]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.490084 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7c57c5ca55f24621b66197a3698504bd. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:50.489722 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 1d10fb4561924e2e8d53c7a0b22a426e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 1d10fb4561924e2e8d53c7a0b22a426e already in progress: creating tablet
I0619 00:11:50.457087 25736 ts_tablet_manager.cc:1382] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.492092 25736 log.cc:986] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/60dfd91c98d94bfbbc6416268c65c378
I0619 00:11:50.492918 25736 ts_tablet_manager.cc:1403] T 60dfd91c98d94bfbbc6416268c65c378 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:50.491137 25082 catalog_manager.cc:3663] Got report from deleted tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:50.493778 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.494431 25741 ts_tablet_manager.cc:1369] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.494457 25071 catalog_manager.cc:2891] Scheduling retry of 1d10fb4561924e2e8d53c7a0b22a426e Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 48 ms (attempt = 2)
I0619 00:11:50.495411 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
W0619 00:11:50.496925 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.491514 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.498337 25067 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 162 ms (attempt = 4)
W0619 00:11:50.498831 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:50.499090 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a52cbcbe35374d518bca1e0fc24d88d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.499070 25067 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 137 ms (attempt = 4)
W0619 00:11:50.489650 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.501536 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7c57c5ca55f24621b66197a3698504bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.478452 25804 consensus_queue.cc:204] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.504835 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.507015 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_95a48b4d8a824265bb7291a68eb33dc3" } new_table_name: "default.table_ce167e3b89b8423a827ae8559f500233" alter_external_catalogs: true
W0619 00:11:50.508235 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 because the tablet was not found. No further retry: Not found: Tablet not found: 3b80a302aacb418091aee54ab4be7249
I0619 00:11:50.508780 25463 ts_tablet_manager.cc:1173] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:50.508927 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:50.509342 25071 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 63 ms (attempt = 2)
I0619 00:11:50.510819 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.511086 25068 catalog_manager.cc:2891] Scheduling retry of aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC with a delay of 298 ms (attempt = 5)
W0619 00:11:50.511354 25068 catalog_manager.cc:2910] Async tablet task aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet aaf62da07a9740c1aa44359d3f604513 consensus state has no leader
I0619 00:11:50.512048 25462 ts_tablet_manager.cc:1173] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.498365 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72e28bc6b57442b782d8aa07e1152cf4" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:50.513366 25419 leader_election.cc:306] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.514375 25727 tablet_bootstrap.cc:427] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:50.514716 25727 ts_tablet_manager.cc:1045] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.127s	user 0.003s	sys 0.003s
I0619 00:11:50.514684 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.515682 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.515985 25727 raft_consensus.cc:325] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.516562 25727 raft_consensus.cc:351] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.516769 25727 raft_consensus.cc:628] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:50.518194 25725 tablet_bootstrap.cc:427] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:50.518539 25725 ts_tablet_manager.cc:1045] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.045s	user 0.004s	sys 0.002s
I0619 00:11:50.518980 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 151 ms (attempt = 4)
W0619 00:11:50.519444 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:50.473306 25776 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.027s (election complete)
I0619 00:11:50.478052 25724 tablet_bootstrap.cc:427] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.489225 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72e28bc6b57442b782d8aa07e1152cf4" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:50.520105 25776 raft_consensus.cc:2487] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:50.520323 25624 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.178s (vote granted)
I0619 00:11:50.519968 25725 raft_consensus.cc:325] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.520566 25725 raft_consensus.cc:351] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.520551 25624 raft_consensus.cc:2245] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:50.520767 25725 raft_consensus.cc:628] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:50.521179 25724 ts_tablet_manager.cc:1045] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.112s	user 0.004s	sys 0.001s
I0619 00:11:50.521220 25725 consensus_queue.cc:227] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.522509 25724 raft_consensus.cc:325] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.523205 25724 raft_consensus.cc:351] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.523394 25724 raft_consensus.cc:628] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:50.523838 25724 consensus_queue.cc:227] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.527148 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet c18355df78d344839a56189c7549ab1f
I0619 00:11:50.527444 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:50.527842 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:50.528221 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:50.528527 25080 catalog_manager.cc:4596] ReplaceTablet: tablet c18355df78d344839a56189c7549ab1f deleted and replaced by tablet 7277051d52dd4be1a6dfa06488c98293
W0619 00:11:50.529007 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.529208 25087 catalog_manager.cc:3663] Got report from deleted tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:50.529855 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:50.530618 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.536597 25082 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 17 ms (attempt = 1)
W0619 00:11:50.536900 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.535699 25809 tablet.cc:1200] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3
I0619 00:11:50.538529 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.517163 25727 consensus_queue.cc:227] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.541296 25601 ts_tablet_manager.cc:1173] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.541638 25725 ts_tablet_manager.cc:1073] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.023s	user 0.003s	sys 0.000s
I0619 00:11:50.542040 25725 ts_tablet_manager.cc:1028] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:50.540431 25325 tablet_service.cc:759] Processing CreateTablet for tablet 4a88e394b199431e8b794a1b1aec847f (table=default.table_dd9523cf3fda45cd8bc544abb9dfba01 [id=9b96421847354087b7a4fa2afc633be3]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.541676 25336 tablet_service.cc:759] Processing CreateTablet for tablet 0a33940e5347489993ab4679e9aeafde (table=default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.543133 25332 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.542234 25333 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.543329 25725 tablet_bootstrap.cc:427] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:50.543264 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a33940e5347489993ab4679e9aeafde. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.545415 25725 tablet_bootstrap.cc:578] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:50.545768 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
W0619 00:11:50.546367 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:50.546609 25067 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 37 ms (attempt = 1)
I0619 00:11:50.542186 25741 ts_tablet_manager.cc:1382] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.547860 25082 catalog_manager.cc:3663] Got report from deleted tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:50.547930 25741 log.cc:986] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/985e23a32fa84104a06dd19d97117ab2
I0619 00:11:50.548532 25741 ts_tablet_manager.cc:1403] T 985e23a32fa84104a06dd19d97117ab2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:50.548691 25331 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:50.542836 25811 raft_consensus.cc:576] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I0619 00:11:50.543509 25730 raft_consensus.cc:2069] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:50.549099 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.550974 25730 ts_tablet_manager.cc:1369] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.554980 25069 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 45 ms (attempt = 2)
W0619 00:11:50.555238 25069 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.554653 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:50.546558 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.560025 25727 ts_tablet_manager.cc:1073] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.045s	user 0.004s	sys 0.000s
W0619 00:11:50.560907 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.561137 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 40 ms (attempt = 2)
I0619 00:11:50.561715 25330 ts_tablet_manager.cc:1173] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.563997 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_aecb1278dc98420aa83aaae47fb56937" } new_table_name: "default.table_4c17e965f551446d965e5fa1499cad9a" alter_external_catalogs: true
I0619 00:11:50.547081 25423 leader_election.cc:379] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.568413 25423 leader_election.cc:270] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:11:50.561463 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:50.570996 25796 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.161s (election complete)
I0619 00:11:50.571244 25796 raft_consensus.cc:2539] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:50.571435 25796 raft_consensus.cc:457] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:50.571650 25796 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.133s (starting election)
I0619 00:11:50.571835 25796 raft_consensus.cc:2784] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:50.573699 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.573913 25326 ts_tablet_manager.cc:1173] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.574460 25323 ts_tablet_manager.cc:1173] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.578052 25796 raft_consensus.cc:479] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.579267 25796 leader_election.cc:243] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.579818 25796 leader_election.cc:243] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.574975 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:50.581099 25727 ts_tablet_manager.cc:1028] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:50.581187 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72e28bc6b57442b782d8aa07e1152cf4" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:50.581449 25727 tablet_bootstrap.cc:427] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:50.581602 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 985e23a32fa84104a06dd19d97117ab2 (table default.table_d946420480b24fe2810193b44fafa15b [id=23387ae786214a72874c9ff28f39db49]) successfully deleted
I0619 00:11:50.581686 25356 raft_consensus.cc:2784] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:50.581976 25356 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.218s (vote granted)
I0619 00:11:50.581694 25071 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 86 ms (attempt = 3)
W0619 00:11:50.582590 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:50.583047 25067 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 98 ms (attempt = 3)
W0619 00:11:50.583431 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 985e23a32fa84104a06dd19d97117ab2 because the tablet was not found. No further retry: Not found: Tablet not found: 985e23a32fa84104a06dd19d97117ab2
W0619 00:11:50.583896 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:50.584118 25067 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 261 ms (attempt = 5)
I0619 00:11:50.584516 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.580871 25082 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 65 ms (attempt = 1)
W0619 00:11:50.585209 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:50.585155 25323 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.548269 25724 ts_tablet_manager.cc:1073] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.027s	user 0.003s	sys 0.000s
I0619 00:11:50.585602 25724 ts_tablet_manager.cc:1028] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.586000 25724 tablet_bootstrap.cc:427] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:50.585997 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:50.585988 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet c18355df78d344839a56189c7549ab1f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c18355df78d344839a56189c7549ab1f already in progress: creating tablet
I0619 00:11:50.586314 25067 catalog_manager.cc:2891] Scheduling retry of c18355df78d344839a56189c7549ab1f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 56 ms (attempt = 1)
W0619 00:11:50.586726 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.586966 25067 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 94 ms (attempt = 3)
I0619 00:11:50.587431 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 277 ms (attempt = 5)
W0619 00:11:50.587990 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
W0619 00:11:50.588320 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.588557 25067 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 151 ms (attempt = 4)
W0619 00:11:50.590513 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.590976 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4a88e394b199431e8b794a1b1aec847f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.591394 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 259 ms (attempt = 5)
W0619 00:11:50.591702 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.592018 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.590310 25336 ts_tablet_manager.cc:1173] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.592094 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72e28bc6b57442b782d8aa07e1152cf4" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:50.592547 25624 raft_consensus.cc:2784] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:50.592824 25624 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.373s (vote granted)
I0619 00:11:50.593148 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.593686 25736 tablet_replica.cc:261] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:50.593958 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:50.594197 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:50.594429 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:50.594655 25725 tablet_bootstrap.cc:427] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:50.594681 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:50.593616 25083 catalog_manager.cc:3786] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "c007a00e65ad41a79da1a0c1050c55b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:50.595074 25083 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:50.595367 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:50.595357 25067 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 1024 ms (attempt = 7)
W0619 00:11:50.595630 25067 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.595685 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:50.595064 25725 ts_tablet_manager.cc:1045] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.052s	user 0.004s	sys 0.002s
I0619 00:11:50.596441 25323 tablet_service.cc:759] Processing CreateTablet for tablet 775f0b6b309c49278066a5e7cf623af2 (table=default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.597010 25624 raft_consensus.cc:2245] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:50.597836 25725 raft_consensus.cc:325] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.598479 25725 raft_consensus.cc:351] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.598728 25725 raft_consensus.cc:628] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:50.599231 25725 consensus_queue.cc:227] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.600422 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 775f0b6b309c49278066a5e7cf623af2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.601116 25069 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 84 ms (attempt = 3)
W0619 00:11:50.601387 25069 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.595945 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:50.608891 25736 raft_consensus.cc:2042] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.610162 25736 raft_consensus.cc:2069] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:50.612757 25356 raft_consensus.cc:2245] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:50.613997 25419 leader_election.cc:379] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.614286 25419 leader_election.cc:270] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:50.614804 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.614070 25736 ts_tablet_manager.cc:1369] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.615294 25796 raft_consensus.cc:2732] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.064s (election complete)
I0619 00:11:50.615602 25796 raft_consensus.cc:2539] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:50.616365 25796 raft_consensus.cc:593] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:50.616842 25796 consensus_queue.cc:204] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.617103 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1d10fb4561924e2e8d53c7a0b22a426e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.619576 25462 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.619669 25725 ts_tablet_manager.cc:1073] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.000s
I0619 00:11:50.620067 25725 ts_tablet_manager.cc:1028] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
W0619 00:11:50.620549 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.621242 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.621965 25423 leader_election.cc:379] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.622817 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.624462 25462 tablet_service.cc:759] Processing CreateTablet for tablet 4a88e394b199431e8b794a1b1aec847f (table=default.table_dd9523cf3fda45cd8bc544abb9dfba01 [id=9b96421847354087b7a4fa2afc633be3]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.588557 25724 tablet_bootstrap.cc:578] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:50.625679 25083 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_62e2a91242c24687b8c59b83aa396c02" }
I0619 00:11:50.627714 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4a88e394b199431e8b794a1b1aec847f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.631801 25727 tablet_bootstrap.cc:578] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:50.632833 25730 ts_tablet_manager.cc:1382] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.633116 25730 log.cc:986] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/292b164659e343158de8e7e3eeb9be31
I0619 00:11:50.633626 25730 ts_tablet_manager.cc:1403] T 292b164659e343158de8e7e3eeb9be31 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:50.639330 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.639863 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
W0619 00:11:50.640314 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet ac129ed44f68432f800420175aa0a630 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ac129ed44f68432f800420175aa0a630 already in progress: creating tablet
I0619 00:11:50.640583 25067 catalog_manager.cc:2891] Scheduling retry of ac129ed44f68432f800420175aa0a630 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 256 ms (attempt = 5)
I0619 00:11:50.641847 25725 tablet_bootstrap.cc:427] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:50.644109 25725 tablet_bootstrap.cc:578] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:50.646114 25462 ts_tablet_manager.cc:1173] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:50.646963 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
W0619 00:11:50.648960 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.651080 25067 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 74 ms (attempt = 2)
W0619 00:11:50.626821 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:50.651278 25068 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 34 ms (attempt = 2)
I0619 00:11:50.651412 25071 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 28 ms (attempt = 1)
W0619 00:11:50.651569 25068 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:50.652571 25601 tablet_service.cc:759] Processing CreateTablet for tablet 4a88e394b199431e8b794a1b1aec847f (table=default.table_dd9523cf3fda45cd8bc544abb9dfba01 [id=9b96421847354087b7a4fa2afc633be3]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.652808 25603 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.653144 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4a88e394b199431e8b794a1b1aec847f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.653568 25603 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:50.654033 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.654284 25071 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 36 ms (attempt = 2)
I0619 00:11:50.655603 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.656008 25602 tablet_service.cc:759] Processing CreateTablet for tablet 0a33940e5347489993ab4679e9aeafde (table=default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.656376 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:50.656545 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a33940e5347489993ab4679e9aeafde. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.656975 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_e248c37d574c4073ad57f7b09e7bf793"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:50.657876 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e248c37d574c4073ad57f7b09e7bf793 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.661911 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.669836 25724 tablet_bootstrap.cc:427] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.670167 25724 ts_tablet_manager.cc:1045] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.084s	user 0.004s	sys 0.001s
I0619 00:11:50.672022 25724 raft_consensus.cc:325] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.672668 25724 raft_consensus.cc:351] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.672906 25724 raft_consensus.cc:628] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:50.673368 25724 consensus_queue.cc:227] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:50.674168 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
I0619 00:11:50.674396 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 23 ms (attempt = 1)
W0619 00:11:50.674728 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:50.674940 25071 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 73 ms (attempt = 3)
I0619 00:11:50.656522 25462 tablet_service.cc:759] Processing CreateTablet for tablet 0a33940e5347489993ab4679e9aeafde (table=default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.675740 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a33940e5347489993ab4679e9aeafde. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:50.676457 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.677759 25082 catalog_manager.cc:3663] Got report from deleted tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (replaced by tablet a85577b0beba4276b963e5c360544bc7): Sending delete request for this tablet
W0619 00:11:50.678233 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.678426 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:50.680975 25724 ts_tablet_manager.cc:1073] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.001s
I0619 00:11:50.681354 25724 ts_tablet_manager.cc:1028] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.681740 25724 tablet_bootstrap.cc:427] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:50.683056 25082 catalog_manager.cc:3663] Got report from deleted tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (replaced by tablet a85577b0beba4276b963e5c360544bc7): Sending delete request for this tablet
I0619 00:11:50.683862 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:50.683990 25724 tablet_bootstrap.cc:578] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:50.686206 25069 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 136 ms (attempt = 4)
W0619 00:11:50.686512 25069 catalog_manager.cc:2910] Async tablet task 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.687052 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.687417 25068 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 89 ms (attempt = 3)
W0619 00:11:50.692245 25068 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:50.695266 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.700625 25809 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 3 got alter request for version 3
I0619 00:11:50.705174 25325 ts_tablet_manager.cc:1173] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.712783 25323 ts_tablet_manager.cc:1173] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.714843 25463 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.715268 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.715932 25470 tablet_service.cc:759] Processing CreateTablet for tablet 775f0b6b309c49278066a5e7cf623af2 (table=default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.716523 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 775f0b6b309c49278066a5e7cf623af2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.718140 25602 ts_tablet_manager.cc:1173] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.718847 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.719826 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.720440 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 1d10fb4561924e2e8d53c7a0b22a426e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.721107 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.722761 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:50.723055 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.723309 25736 ts_tablet_manager.cc:1382] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.723445 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.723549 25736 log.cc:986] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/4c0614e9ed96420886243a6868f2e75e
I0619 00:11:50.723855 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.724042 25736 ts_tablet_manager.cc:1403] T 4c0614e9ed96420886243a6868f2e75e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:50.726028 25470 ts_tablet_manager.cc:1173] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.726028 25601 ts_tablet_manager.cc:1173] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.727988 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.728888 25725 tablet_bootstrap.cc:427] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:50.729233 25725 ts_tablet_manager.cc:1045] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.109s	user 0.003s	sys 0.002s
I0619 00:11:50.730859 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.731076 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.730870 25725 raft_consensus.cc:325] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.731657 25725 raft_consensus.cc:351] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.731878 25725 raft_consensus.cc:628] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:50.729593 25087 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]). Expected version 2 got 0
I0619 00:11:50.735023 25087 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 60 ms (attempt = 1)
W0619 00:11:50.735293 25087 catalog_manager.cc:3945] Failed to send 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 72e28bc6b57442b782d8aa07e1152cf4 consensus state has no leader
I0619 00:11:50.736224 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.736760 25330 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.738596 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.739352 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_15724e9cddcb457a8edaebf78d6afc15" }
W0619 00:11:50.741065 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times.
I0619 00:11:50.731577 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.741459 25601 tablet_service.cc:759] Processing CreateTablet for tablet 775f0b6b309c49278066a5e7cf623af2 (table=default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.741711 25603 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.742352 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.742821 25730 tablet_replica.cc:261] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:50.742812 25604 tablet_service.cc:759] Processing CreateTablet for tablet 7277051d52dd4be1a6dfa06488c98293 (table=default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.743119 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:50.742374 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.743333 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:50.743544 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.743728 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:50.743923 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:50.744138 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:50.744335 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:50.744519 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:50.744712 25730 raft_consensus.cc:2042] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.744972 25730 raft_consensus.cc:2069] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:50.743459 25602 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.742513 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 775f0b6b309c49278066a5e7cf623af2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.745805 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 520 ms (attempt = 6)
W0619 00:11:50.746121 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:50.746515 25604 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7277051d52dd4be1a6dfa06488c98293. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.746484 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 292 ms (attempt = 5)
W0619 00:11:50.746815 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:50.746811 25730 ts_tablet_manager.cc:1369] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.747532 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c771d75362c64504aed46e585f8fbdf3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:50.748399 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c771d75362c64504aed46e585f8fbdf3 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.751040 25724 tablet_bootstrap.cc:427] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.751368 25724 ts_tablet_manager.cc:1045] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.070s	user 0.005s	sys 0.007s
I0619 00:11:50.752796 25724 raft_consensus.cc:325] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.753432 25724 raft_consensus.cc:351] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.753666 25724 raft_consensus.cc:628] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:50.754377 25082 catalog_manager.cc:3786] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:50.755358 25082 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]). Expected version 2 got 0
I0619 00:11:50.755761 25724 consensus_queue.cc:227] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.760321 25736 tablet_replica.cc:261] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:50.760607 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.760871 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.764003 25080 catalog_manager.cc:3663] Got report from deleted tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b): Sending delete request for this tablet
I0619 00:11:50.765087 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
W0619 00:11:50.769793 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.771677 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 292b164659e343158de8e7e3eeb9be31 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) successfully deleted
W0619 00:11:50.737525 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.772763 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.772852 25326 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.774158 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.773808 25470 tablet_service.cc:759] Processing CreateTablet for tablet 7277051d52dd4be1a6dfa06488c98293 (table=default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:50.775743 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 292b164659e343158de8e7e3eeb9be31 already in progress: creating tablet
I0619 00:11:50.775969 25067 catalog_manager.cc:2891] Scheduling retry of 292b164659e343158de8e7e3eeb9be31 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 296 ms (attempt = 5)
I0619 00:11:50.776244 25325 tablet_service.cc:759] Processing CreateTablet for tablet 7277051d52dd4be1a6dfa06488c98293 (table=default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.776290 25080 catalog_manager.cc:3663] Got report from deleted tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b): Sending delete request for this tablet
W0619 00:11:50.776490 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.776839 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7277051d52dd4be1a6dfa06488c98293. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.777037 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
W0619 00:11:50.777971 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:50.778208 25067 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 266 ms (attempt = 5)
I0619 00:11:50.733585 25725 consensus_queue.cc:227] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:50.778578 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet c18355df78d344839a56189c7549ab1f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c18355df78d344839a56189c7549ab1f already in progress: creating tablet
I0619 00:11:50.778723 25796 raft_consensus.cc:457] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.778817 25067 catalog_manager.cc:2891] Scheduling retry of c18355df78d344839a56189c7549ab1f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 58 ms (attempt = 2)
I0619 00:11:50.778959 25796 raft_consensus.cc:2732] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.520s (starting election)
W0619 00:11:50.779202 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.779021 25724 ts_tablet_manager.cc:1073] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.027s	user 0.003s	sys 0.001s
I0619 00:11:50.779417 25067 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 270 ms (attempt = 5)
I0619 00:11:50.779232 25796 raft_consensus.cc:479] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.779984 25724 ts_tablet_manager.cc:1028] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.780416 25796 leader_election.cc:243] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.780992 25724 tablet_bootstrap.cc:427] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:50.781289 25796 leader_election.cc:243] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:11:50.783061 25748 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I0619 00:11:50.783193 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "292b164659e343158de8e7e3eeb9be31" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:50.783427 25725 ts_tablet_manager.cc:1073] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.054s	user 0.003s	sys 0.000s
W0619 00:11:50.783782 25748 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:11:50.784935 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.785547 25419 leader_election.cc:306] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:50.786671 25423 leader_election.cc:306] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 292b164659e343158de8e7e3eeb9be31
I0619 00:11:50.786928 25423 leader_election.cc:270] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:50.787662 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.788043 25796 raft_consensus.cc:2732] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.966s (election complete)
I0619 00:11:50.788125 25325 ts_tablet_manager.cc:1173] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.788278 25796 raft_consensus.cc:2487] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:50.788759 25068 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 151 ms (attempt = 4)
W0619 00:11:50.789033 25068 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:50.789399 25819 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 3 got alter request for version 3
I0619 00:11:50.790153 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.791110 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_2e9154601fc34c21a4a6c3e101a644dc"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:50.791749 25323 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:50.792100 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2e9154601fc34c21a4a6c3e101a644dc in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:50.795521 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.796170 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.798902 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 53688f0e4a604bc4a07dec9278a8dad6
I0619 00:11:50.798861 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.799188 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 53688f0e4a604bc4a07dec9278a8dad6 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.799563 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 53688f0e4a604bc4a07dec9278a8dad6 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.799995 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 53688f0e4a604bc4a07dec9278a8dad6 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.800945 25730 ts_tablet_manager.cc:1382] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:11:50.800988 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.801225 25067 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 165 ms (attempt = 4)
I0619 00:11:50.801618 25730 log.cc:986] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/70b9c8c4af0945b69b5577e47b61c054
W0619 00:11:50.801975 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:50.802188 25727 tablet_bootstrap.cc:427] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:50.802192 25067 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 143 ms (attempt = 4)
I0619 00:11:50.801352 25724 tablet_bootstrap.cc:578] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:50.802512 25727 ts_tablet_manager.cc:1045] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.221s	user 0.006s	sys 0.011s
I0619 00:11:50.802880 25811 raft_consensus.cc:457] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.803123 25811 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.745s (starting election)
W0619 00:11:50.803593 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.803361 25811 raft_consensus.cc:479] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.804376 25811 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.804154 25727 raft_consensus.cc:325] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.804795 25727 raft_consensus.cc:351] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.804797 25811 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.804978 25727 raft_consensus.cc:628] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:50.805371 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.805423 25727 consensus_queue.cc:227] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.806319 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.806340 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:50.808326 25730 ts_tablet_manager.cc:1403] T 70b9c8c4af0945b69b5577e47b61c054 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:11:50.808881 25553 leader_election.cc:306] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.810189 25819 tablet.cc:1195] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 3 got alter request for version 3
I0619 00:11:50.813758 25068 catalog_manager.cc:2891] Scheduling retry of aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC with a delay of 560 ms (attempt = 6)
W0619 00:11:50.814030 25068 catalog_manager.cc:2910] Async tablet task aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet aaf62da07a9740c1aa44359d3f604513 consensus state has no leader
I0619 00:11:50.814860 25604 ts_tablet_manager.cc:1173] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.815397 25624 raft_consensus.cc:1081] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:50.815965 25463 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:50.818908 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.819061 25727 ts_tablet_manager.cc:1073] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
I0619 00:11:50.819605 25727 ts_tablet_manager.cc:1028] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:50.820165 25727 tablet_bootstrap.cc:427] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:50.824314 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.824597 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.824846 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(1d10fb4561924e2e8d53c7a0b22a426e)
I0619 00:11:50.825054 25736 raft_consensus.cc:2042] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.825347 25736 raft_consensus.cc:2069] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:50.826174 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7f60ea5c5def47ee846d0186bbc90c18 already in progress: creating tablet
I0619 00:11:50.826436 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 37 ms (attempt = 1)
I0619 00:11:50.770357 25462 ts_tablet_manager.cc:1173] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:50.827046 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:50.827297 25067 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 84 ms (attempt = 3)
W0619 00:11:50.827529 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:50.827797 25071 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 73 ms (attempt = 3)
I0619 00:11:50.828011 25724 tablet_bootstrap.cc:427] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.828353 25724 ts_tablet_manager.cc:1045] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.047s	user 0.003s	sys 0.002s
I0619 00:11:50.828712 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.829167 25602 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:50.829422 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.829396 25603 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.829954 25724 raft_consensus.cc:325] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.830621 25724 raft_consensus.cc:351] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.830667 25736 ts_tablet_manager.cc:1369] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.830863 25724 raft_consensus.cc:628] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:50.831620 25601 ts_tablet_manager.cc:1173] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.831524 25724 consensus_queue.cc:227] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.831651 25600 tablet_service.cc:759] Processing CreateTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb (table=default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.832801 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0474be659e5e4a4cbfc181b8c812b1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.833684 25725 ts_tablet_manager.cc:1028] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:50.834218 25725 tablet_bootstrap.cc:427] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:50.834846 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.835839 25604 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.836778 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7277051d52dd4be1a6dfa06488c98293. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.836891 25489 raft_consensus.cc:1081] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:50.837056 25725 tablet_bootstrap.cc:578] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:50.839365 25083 catalog_manager.cc:3663] Got report from deleted tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:50.840090 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.842734 25807 raft_consensus.cc:457] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.842974 25807 raft_consensus.cc:2732] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.610s (starting election)
I0619 00:11:50.843228 25807 raft_consensus.cc:479] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.844211 25807 leader_election.cc:243] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.844630 25807 leader_election.cc:243] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.844940 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:50.845424 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:50.845705 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.844992 25727 tablet_bootstrap.cc:578] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:50.847136 25083 catalog_manager.cc:3663] Got report from deleted tablet 292b164659e343158de8e7e3eeb9be31 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0): Sending delete request for this tablet
I0619 00:11:50.847812 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:50.850328 25807 consensus_queue.cc:963] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:50.851817 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.852211 25083 catalog_manager.cc:3663] Got report from deleted tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) (Table deleted at 2018-06-19 00:11:48 PDT): Sending delete request for this tablet
I0619 00:11:50.853052 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 89a7ff9cdbc046779799e4e8f71adb42 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:48 PDT)
I0619 00:11:50.854861 25323 tablet_service.cc:759] Processing CreateTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb (table=default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.855396 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0474be659e5e4a4cbfc181b8c812b1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.854315 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:50.858471 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ef7e3bdd5d4043abbae5282cce9c6fd7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:50.859407 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:50.863864 25323 ts_tablet_manager.cc:1173] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.868554 25730 tablet_replica.cc:261] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:50.868855 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:50.869089 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:50.869302 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:50.869514 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:50.869777 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:50.869988 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:50.870198 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:50.870383 25730 raft_consensus.cc:2042] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.870666 25730 raft_consensus.cc:2069] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:50.872207 25730 ts_tablet_manager.cc:1369] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:50.861114 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 because the tablet was not found. No further retry: Not found: Tablet not found: 292b164659e343158de8e7e3eeb9be31
I0619 00:11:50.879122 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.884479 25080 catalog_manager.cc:3663] Got report from deleted tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:50.885110 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.888605 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 1037 ms (attempt = 7)
W0619 00:11:50.888880 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:50.889340 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet 72e28bc6b57442b782d8aa07e1152cf4 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:50.891252 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.891480 25071 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 64 ms (attempt = 3)
I0619 00:11:50.892609 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_6eabaa2225f44436880ea396f6bc6acd" } new_table_name: "default.table_fb89f806be98424aa43bd01b3f816a80" alter_external_catalogs: true
I0619 00:11:50.897400 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:50.899492 25727 tablet_bootstrap.cc:427] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:50.899864 25727 ts_tablet_manager.cc:1045] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.080s	user 0.005s	sys 0.002s
I0619 00:11:50.901808 25727 raft_consensus.cc:325] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.902700 25727 raft_consensus.cc:351] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.902984 25727 raft_consensus.cc:628] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:50.903694 25727 consensus_queue.cc:227] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:50.906276 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.909489 25807 consensus_queue.cc:963] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:50.911418 25083 catalog_manager.cc:3663] Got report from deleted tablet ac129ed44f68432f800420175aa0a630 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (replaced by tablet 35acf55f35064c1491122a13ff9f5744): Sending delete request for this tablet
I0619 00:11:50.912070 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:50.913563 25727 ts_tablet_manager.cc:1073] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I0619 00:11:50.913918 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:50.914193 25727 ts_tablet_manager.cc:1028] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:50.914336 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:50.914556 25067 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 288 ms (attempt = 5)
I0619 00:11:50.914563 25325 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:50.915496 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.916111 25290 leader_election.cc:306] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:50.917995 25083 catalog_manager.cc:3663] Got report from deleted tablet ac129ed44f68432f800420175aa0a630 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (replaced by tablet 35acf55f35064c1491122a13ff9f5744): Sending delete request for this tablet
I0619 00:11:50.918668 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ac129ed44f68432f800420175aa0a630 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 35acf55f35064c1491122a13ff9f5744)
I0619 00:11:50.918779 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.918784 25462 tablet_service.cc:759] Processing CreateTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb (table=default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.919384 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0474be659e5e4a4cbfc181b8c812b1eb. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:50.923039 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet c18355df78d344839a56189c7549ab1f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c18355df78d344839a56189c7549ab1f already in progress: creating tablet
I0619 00:11:50.923285 25071 catalog_manager.cc:2891] Scheduling retry of c18355df78d344839a56189c7549ab1f Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 52 ms (attempt = 1)
I0619 00:11:50.924398 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.925753 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 292b164659e343158de8e7e3eeb9be31 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:50.925879 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_0a0cd847ea574c2eb82ae9dfaf67e95f" } new_table_name: "default.table_c74748783ca3450eb3c5c3397e03d506" alter_external_catalogs: true
I0619 00:11:50.929460 25082 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 20 ms (attempt = 1)
W0619 00:11:50.929728 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:50.930568 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:50.930884 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:50.931335 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:50.932893 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:50.933229 25080 catalog_manager.cc:4596] ReplaceTablet: tablet 72e28bc6b57442b782d8aa07e1152cf4 deleted and replaced by tablet d62715e230154e2abe126ef97feec7ce
W0619 00:11:50.881829 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.887977 25724 ts_tablet_manager.cc:1073] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.059s	user 0.004s	sys 0.000s
I0619 00:11:50.892594 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.893085 25604 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.894771 25602 tablet_service.cc:759] Processing CreateTablet for tablet a4c653a255984d3bac88df63362b9334 (table=default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.895153 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.895869 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.897073 25811 raft_consensus.cc:457] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.914916 25727 tablet_bootstrap.cc:427] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:50.941205 25811 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.658s (starting election)
I0619 00:11:50.930393 25598 tablet_service.cc:759] Processing CreateTablet for tablet 3e7724f967ae41d7b523055d1a4f58db (table=default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:50.941469 25811 raft_consensus.cc:479] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.942473 25811 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:50.942895 25811 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.943882 25068 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 264 ms (attempt = 5)
W0619 00:11:50.944128 25068 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:50.898092 25600 ts_tablet_manager.cc:1173] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.944631 25724 ts_tablet_manager.cc:1028] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.944979 25724 tablet_bootstrap.cc:427] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:50.945403 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4c653a255984d3bac88df63362b9334. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.947175 25724 tablet_bootstrap.cc:578] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:50.948267 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3e7724f967ae41d7b523055d1a4f58db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:50.948917 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03522c48c50943b38cae5aa88562e583" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:50.950706 25553 leader_election.cc:306] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.951560 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:50.952091 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.953325 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.957314 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.957558 25602 ts_tablet_manager.cc:1173] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.957669 25600 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:50.957644 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:50.961771 25558 leader_election.cc:306] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.962024 25558 leader_election.cc:270] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:50.961989 25068 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 38 ms (attempt = 2)
W0619 00:11:50.962246 25068 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:50.962774 25736 ts_tablet_manager.cc:1382] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:50.963001 25811 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.943s (election complete)
I0619 00:11:50.963225 25811 raft_consensus.cc:2487] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:50.958200 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:50.965713 25727 tablet_bootstrap.cc:578] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:50.969760 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.970542 25598 ts_tablet_manager.cc:1173] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.976119 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 89a7ff9cdbc046779799e4e8f71adb42 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:48 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:50.976546 25741 tablet_replica.cc:261] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:50.976794 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:50.976778 25724 tablet_bootstrap.cc:427] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:50.976984 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:50.977159 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:50.977174 25724 ts_tablet_manager.cc:1045] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.032s	user 0.005s	sys 0.003s
I0619 00:11:50.977335 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:50.977532 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:50.977728 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:50.977891 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(89a7ff9cdbc046779799e4e8f71adb42)
I0619 00:11:50.978044 25741 raft_consensus.cc:2042] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:50.978284 25741 raft_consensus.cc:2069] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:50.978752 25724 raft_consensus.cc:325] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.979357 25724 raft_consensus.cc:351] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:50.979568 25724 raft_consensus.cc:628] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:50.979944 25741 ts_tablet_manager.cc:1369] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:50.980147 25724 consensus_queue.cc:227] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:50.982730 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:50.984460 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:50.984599 25736 log.cc:986] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/1d10fb4561924e2e8d53c7a0b22a426e
I0619 00:11:50.985114 25736 ts_tablet_manager.cc:1403] T 1d10fb4561924e2e8d53c7a0b22a426e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:50.960145 25470 ts_tablet_manager.cc:1173] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:50.985553 25724 ts_tablet_manager.cc:1073] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I0619 00:11:50.967233 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:50.985949 25356 raft_consensus.cc:1081] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:50.986065 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 91 ms (attempt = 3)
I0619 00:11:50.986321 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:50.985952 25724 ts_tablet_manager.cc:1028] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:50.987490 25724 tablet_bootstrap.cc:427] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:50.987434 25775 consensus_queue.cc:963] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:50.989199 25775 raft_consensus.cc:457] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:50.989439 25775 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.125s (starting election)
I0619 00:11:50.989722 25775 raft_consensus.cc:479] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:50.990753 25775 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:50.991281 25775 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:50.992810 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:50.994765 25083 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 26 ms (attempt = 1)
W0619 00:11:50.995049 25083 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
W0619 00:11:50.995033 25419 leader_election.cc:306] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:50.995522 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:50.998987 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet c18355df78d344839a56189c7549ab1f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c18355df78d344839a56189c7549ab1f already in progress: creating tablet
I0619 00:11:50.999222 25067 catalog_manager.cc:2891] Scheduling retry of c18355df78d344839a56189c7549ab1f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 95 ms (attempt = 3)
W0619 00:11:50.999650 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 53688f0e4a604bc4a07dec9278a8dad6 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 53688f0e4a604bc4a07dec9278a8dad6 already in progress: creating tablet
I0619 00:11:50.999851 25067 catalog_manager.cc:2891] Scheduling retry of 53688f0e4a604bc4a07dec9278a8dad6 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 35 ms (attempt = 1)
I0619 00:11:50.986382 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 60dfd91c98d94bfbbc6416268c65c378 (table default.table_041edc89d445491abcd8368e840c196f [id=e5be765de16c4735a50c3acb4a261add]) successfully deleted
I0619 00:11:51.001572 25068 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 92 ms (attempt = 3)
W0619 00:11:51.001862 25068 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.002276 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:51.003311 25083 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:51.003151 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b1931959ceee4a6caa4a431f2c8234ba"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:51.004050 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b1931959ceee4a6caa4a431f2c8234ba in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:51.008430 25741 ts_tablet_manager.cc:1382] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.008728 25741 log.cc:986] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:51.010669 25622 raft_consensus.cc:1081] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:51.010989 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.010879 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:51.011330 25624 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.053s (vote granted)
I0619 00:11:51.011541 25624 raft_consensus.cc:2245] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:51.012106 25470 tablet_service.cc:759] Processing CreateTablet for tablet a4c653a255984d3bac88df63362b9334 (table=default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.013083 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4c653a255984d3bac88df63362b9334. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.014824 25724 tablet_bootstrap.cc:578] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.018765 25736 tablet_replica.cc:261] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:51.019073 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.019273 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.019471 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.019693 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.019927 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.021404 25725 tablet_bootstrap.cc:427] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.021785 25725 ts_tablet_manager.cc:1045] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.188s	user 0.006s	sys 0.007s
I0619 00:11:51.022439 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 73 ms (attempt = 2)
W0619 00:11:51.022733 25067 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.023178 25725 raft_consensus.cc:325] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.023823 25725 raft_consensus.cc:351] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.024019 25725 raft_consensus.cc:628] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:51.024822 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
I0619 00:11:51.025112 25462 ts_tablet_manager.cc:1173] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.025306 25602 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.025902 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.026077 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03522c48c50943b38cae5aa88562e583" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:51.030023 25600 tablet_service.cc:759] Processing CreateTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 (table=default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.030652 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 86959e7dda374ba9ac23d3c1fd30d993. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.024435 25725 consensus_queue.cc:227] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.034498 25080 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.035218 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 53688f0e4a604bc4a07dec9278a8dad6 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.035142 25082 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:51.035466 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.036077 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 62 ms (attempt = 1)
I0619 00:11:51.037075 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_48a6321c18954a77805c579d820e8d00" }
I0619 00:11:51.038148 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) successfully deleted
W0619 00:11:51.038652 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:51.038909 25071 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 56 ms (attempt = 2)
I0619 00:11:51.039295 25725 ts_tablet_manager.cc:1073] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I0619 00:11:51.040805 25087 catalog_manager.cc:3663] Got report from deleted tablet 3b80a302aacb418091aee54ab4be7249 (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet acc89522b693498c987ca5d8ef8275ea): Sending delete request for this tablet
I0619 00:11:51.041474 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:51.042333 25080 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.043123 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.047412 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:51.048995 25083 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]). Expected version 2 got 0
I0619 00:11:51.049769 25083 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 24 ms (attempt = 1)
W0619 00:11:51.050046 25083 catalog_manager.cc:3945] Failed to send acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.050771 25741 ts_tablet_manager.cc:1403] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:51.032901 25819 tablet.cc:1200] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:51.040680 25807 raft_consensus.cc:457] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.051075 25807 raft_consensus.cc:2732] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.728s (starting election)
I0619 00:11:51.051345 25807 raft_consensus.cc:479] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.052042 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.052395 25807 leader_election.cc:243] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.050704 25727 tablet_bootstrap.cc:427] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:51.053086 25727 ts_tablet_manager.cc:1045] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.138s	user 0.007s	sys 0.000s
I0619 00:11:51.052809 25807 leader_election.cc:243] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.053678 25775 consensus_queue.cc:963] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:11:51.009974 25290 leader_election.cc:306] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 89a7ff9cdbc046779799e4e8f71adb42
I0619 00:11:51.054179 25290 leader_election.cc:270] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:51.054569 25290 raft_consensus.cc:2442] T 89a7ff9cdbc046779799e4e8f71adb42 P c007a00e65ad41a79da1a0c1050c55b4: Unable to run election callback: Service unavailable: Thread pool token was shut down
W0619 00:11:51.054924 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 44973.7us
W0619 00:11:51.055215 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.055822 25325 tablet_service.cc:759] Processing CreateTablet for tablet a4c653a255984d3bac88df63362b9334 (table=default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.056390 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4c653a255984d3bac88df63362b9334. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:51.056291 25558 leader_election.cc:306] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.056417 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.056571 25558 leader_election.cc:270] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.057245 25811 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.071s (election complete)
I0619 00:11:51.057355 25423 leader_election.cc:379] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.057459 25811 raft_consensus.cc:2487] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:51.057600 25423 leader_election.cc:270] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:51.058130 25462 tablet_service.cc:759] Processing CreateTablet for tablet 3e7724f967ae41d7b523055d1a4f58db (table=default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.059607 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3e7724f967ae41d7b523055d1a4f58db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.061774 25083 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.062465 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
W0619 00:11:51.062948 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.063486 25725 ts_tablet_manager.cc:1028] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:51.064257 25725 tablet_bootstrap.cc:427] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:51.054680 25727 raft_consensus.cc:325] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.064970 25727 raft_consensus.cc:351] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.065179 25727 raft_consensus.cc:628] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:51.065801 25325 ts_tablet_manager.cc:1173] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.068729 25600 ts_tablet_manager.cc:1173] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.072233 25725 tablet_bootstrap.cc:578] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:51.073251 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:51.074654 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.075652 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4c0614e9ed96420886243a6868f2e75e already in progress: creating tablet
I0619 00:11:51.075912 25067 catalog_manager.cc:2891] Scheduling retry of 4c0614e9ed96420886243a6868f2e75e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 522 ms (attempt = 6)
I0619 00:11:51.078240 25068 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 33 ms (attempt = 2)
I0619 00:11:51.069820 25724 tablet_bootstrap.cc:427] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
W0619 00:11:51.078511 25068 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.078897 25724 ts_tablet_manager.cc:1045] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.092s	user 0.004s	sys 0.001s
W0619 00:11:51.079651 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 70b9c8c4af0945b69b5577e47b61c054 because the tablet was not found. No further retry: Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054
W0619 00:11:51.080191 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
I0619 00:11:51.080449 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 31 ms (attempt = 1)
I0619 00:11:51.080469 25598 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:51.081490 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
I0619 00:11:51.081760 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 71 ms (attempt = 2)
W0619 00:11:51.082131 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 60dfd91c98d94bfbbc6416268c65c378 because the tablet was not found. No further retry: Not found: Tablet not found: 60dfd91c98d94bfbbc6416268c65c378
I0619 00:11:51.081782 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_531af4913e434ddc8afce2aa7b081d1c"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:51.081707 25724 raft_consensus.cc:325] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.082636 25724 raft_consensus.cc:351] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:51.082882 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_531af4913e434ddc8afce2aa7b081d1c in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:51.082852 25724 raft_consensus.cc:628] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:51.077777 25730 ts_tablet_manager.cc:1382] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.083400 25724 consensus_queue.cc:227] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.083997 25730 log.cc:986] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/f372b9fb174844eca1662c62039a3c71
I0619 00:11:51.084552 25730 ts_tablet_manager.cc:1403] T f372b9fb174844eca1662c62039a3c71 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:51.088449 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet aaf62da07a9740c1aa44359d3f604513
I0619 00:11:51.088747 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aaf62da07a9740c1aa44359d3f604513 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.089116 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aaf62da07a9740c1aa44359d3f604513 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.089452 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aaf62da07a9740c1aa44359d3f604513 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.077733 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.083336 25775 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.949s (election complete)
I0619 00:11:51.084837 25468 tablet_service.cc:759] Processing CreateTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 (table=default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.091429 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.091811 25736 raft_consensus.cc:2042] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.092109 25736 raft_consensus.cc:2069] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.091393 25775 raft_consensus.cc:2539] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:51.092572 25775 raft_consensus.cc:457] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:51.094799 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
W0619 00:11:51.086948 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.098132 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 90 ms (attempt = 3)
I0619 00:11:51.081013 25837 tablet.cc:1200] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
W0619 00:11:51.099761 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.097504 25068 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 168 ms (attempt = 4)
W0619 00:11:51.100291 25068 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
W0619 00:11:51.098374 25067 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.103829 25727 consensus_queue.cc:227] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.105751 25082 catalog_manager.cc:3663] Got report from deleted tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.106393 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:51.108830 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.082870 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 529 ms (attempt = 6)
W0619 00:11:51.110373 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
W0619 00:11:51.112308 25748 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:11:51.112596 25748 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
I0619 00:11:51.112520 25598 tablet_service.cc:759] Processing CreateTablet for tablet d62715e230154e2abe126ef97feec7ce (table=default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:51.112944 25748 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:11:51.112943 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:51.113107 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d62715e230154e2abe126ef97feec7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.113158 25071 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 48 ms (attempt = 1)
W0619 00:11:51.117699 25748 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:11:51.117990 25748 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:11:51.091101 25470 ts_tablet_manager.cc:1173] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.120292 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 86959e7dda374ba9ac23d3c1fd30d993. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.117354 25725 tablet_bootstrap.cc:427] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.121045 25725 ts_tablet_manager.cc:1045] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.057s	user 0.006s	sys 0.000s
I0619 00:11:51.122387 25725 raft_consensus.cc:325] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.092859 25775 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.722s (starting election)
I0619 00:11:51.123040 25725 raft_consensus.cc:351] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.123797 25598 ts_tablet_manager.cc:1173] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:51.123956 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.123093 25775 raft_consensus.cc:2784] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.127475 25819 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:51.126916 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:51.116403 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c771d75362c64504aed46e585f8fbdf3" }
I0619 00:11:51.093606 25736 ts_tablet_manager.cc:1369] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.125627 25083 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.125898 25080 catalog_manager.cc:3663] Got report from deleted tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.128934 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.130372 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.113612 25068 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 102 ms (attempt = 3)
I0619 00:11:51.132401 25730 tablet_replica.cc:261] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
W0619 00:11:51.132553 25068 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.133702 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.133951 25463 tablet_service.cc:759] Processing CreateTablet for tablet d62715e230154e2abe126ef97feec7ce (table=default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.134567 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d62715e230154e2abe126ef97feec7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.135937 25325 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.136454 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.138442 25741 tablet_replica.cc:261] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:51.138702 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:51.138897 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:51.139076 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:51.139261 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:51.139470 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:51.139662 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:51.139865 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(ac129ed44f68432f800420175aa0a630)
I0619 00:11:51.140036 25741 raft_consensus.cc:2042] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.140290 25741 raft_consensus.cc:2069] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.141340 25727 ts_tablet_manager.cc:1073] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.088s	user 0.004s	sys 0.000s
I0619 00:11:51.141747 25727 ts_tablet_manager.cc:1028] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:51.141860 25741 ts_tablet_manager.cc:1369] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.142176 25727 tablet_bootstrap.cc:427] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:51.144276 25727 tablet_bootstrap.cc:578] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:51.087874 25724 ts_tablet_manager.cc:1073] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I0619 00:11:51.144707 25724 ts_tablet_manager.cc:1028] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:51.145082 25724 tablet_bootstrap.cc:427] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:51.146045 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.146477 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.146915 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.147305 25724 tablet_bootstrap.cc:578] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.153404 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 89a7ff9cdbc046779799e4e8f71adb42 (table default.table_76fe64ca5bbd44279ae948d8df882d65 [id=f25f5cc3e1d64c95a0dcc4e36416c416]) successfully deleted
W0619 00:11:51.154937 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7f60ea5c5def47ee846d0186bbc90c18 already in progress: creating tablet
W0619 00:11:51.155210 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.155339 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 81 ms (attempt = 2)
I0619 00:11:51.155463 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:51.156428 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:51.156672 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:51.156963 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:51.157233 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:51.164702 25080 catalog_manager.cc:3663] Got report from deleted tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.165033 25080 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.165652 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:51.165913 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:51.165989 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.166123 25730 raft_consensus.cc:2042] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.166362 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.166471 25730 raft_consensus.cc:2069] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.150455 25325 tablet_service.cc:759] Processing CreateTablet for tablet 3e7724f967ae41d7b523055d1a4f58db (table=default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.167495 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3e7724f967ae41d7b523055d1a4f58db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.167425 25083 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef] alter complete (version 2)
I0619 00:11:51.169394 25468 ts_tablet_manager.cc:1173] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.177445 25325 ts_tablet_manager.cc:1173] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.123258 25725 raft_consensus.cc:628] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:51.178206 25725 consensus_queue.cc:227] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.180186 25725 ts_tablet_manager.cc:1073] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.059s	user 0.004s	sys 0.000s
I0619 00:11:51.180572 25725 ts_tablet_manager.cc:1028] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:51.180994 25725 tablet_bootstrap.cc:427] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:51.181066 25290 leader_election.cc:306] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: ac129ed44f68432f800420175aa0a630
I0619 00:11:51.181191 25463 ts_tablet_manager.cc:1173] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.183691 25725 tablet_bootstrap.cc:578] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:51.185712 25080 catalog_manager.cc:3663] Got report from deleted tablet aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:51.186390 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aaf62da07a9740c1aa44359d3f604513 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.187861 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet a4c653a255984d3bac88df63362b9334
I0619 00:11:51.188192 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:51.188755 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.188755 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.157330 25842 tablet.cc:1200] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:51.189232 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.190191 25730 ts_tablet_manager.cc:1369] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.191126 25083 catalog_manager.cc:3663] Got report from deleted tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.192211 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc22ad98ed2e4e3eb2c62360626a6f87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.193182 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 133 ms (attempt = 4)
W0619 00:11:51.193440 25067 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
W0619 00:11:51.194255 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.195163 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.195221 25290 leader_election.cc:306] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: ac129ed44f68432f800420175aa0a630
I0619 00:11:51.195471 25290 leader_election.cc:270] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:51.195875 25290 raft_consensus.cc:2442] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:11:51.196460 25724 tablet_bootstrap.cc:427] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:51.196804 25724 ts_tablet_manager.cc:1045] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.052s	user 0.003s	sys 0.001s
I0619 00:11:51.200192 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.200335 25736 ts_tablet_manager.cc:1382] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.199923 25325 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.203447 25811 raft_consensus.cc:457] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.203688 25811 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.968s (starting election)
I0619 00:11:51.204339 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.203920 25811 raft_consensus.cc:479] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.205240 25811 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.205626 25811 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.210755 25844 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:51.212832 25083 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:51.203910 25724 raft_consensus.cc:325] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.213282 25724 raft_consensus.cc:351] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.213552 25724 raft_consensus.cc:628] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:51.214262 25724 consensus_queue.cc:227] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.211833 25068 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 512 ms (attempt = 6)
W0619 00:11:51.215847 25068 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:51.217159 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_7937f0a6914a40b8baff07c0983cb44c"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:51.219310 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7937f0a6914a40b8baff07c0983cb44c in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:51.220858 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:51.221092 25071 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 44 ms (attempt = 2)
I0619 00:11:51.222853 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb57d196f0724247b6a7f9a34be8377a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:51.226825 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.227740 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.235568 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.235990 25553 leader_election.cc:306] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.235289 25068 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 135 ms (attempt = 4)
W0619 00:11:51.237757 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.237757 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:51.237231 25068 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.239240 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.240101 25323 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.202375 25841 raft_consensus.cc:457] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:51.207710 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.240486 25841 raft_consensus.cc:2732] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.936s (starting election)
I0619 00:11:51.240808 25841 raft_consensus.cc:479] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.242297 25841 leader_election.cc:243] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.242738 25841 leader_election.cc:243] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.243484 25736 log.cc:986] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/292b164659e343158de8e7e3eeb9be31
I0619 00:11:51.243660 25325 tablet_service.cc:800] Processing DeleteTablet for tablet ac129ed44f68432f800420175aa0a630 with delete_type TABLET_DATA_DELETED (replaced by tablet 35acf55f35064c1491122a13ff9f5744) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.245548 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.245115 25736 ts_tablet_manager.cc:1403] T 292b164659e343158de8e7e3eeb9be31 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:51.246261 25724 ts_tablet_manager.cc:1073] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.049s	user 0.005s	sys 0.000s
I0619 00:11:51.246642 25724 ts_tablet_manager.cc:1028] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
W0619 00:11:51.246646 25423 leader_election.cc:306] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 3b80a302aacb418091aee54ab4be7249
I0619 00:11:51.192309 25462 ts_tablet_manager.cc:1173] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.247144 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb57d196f0724247b6a7f9a34be8377a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:51.249770 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet fb8e94875a8347a0a473c95a77c38795 (table default.table_84d38522d00a488c82a136b79665c4cd [id=41f48f0e1f87464eb43c5bf2a36f70ef]). Expected version 2 got 0
I0619 00:11:51.246976 25724 tablet_bootstrap.cc:427] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:51.253854 25819 tablet.cc:1195] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 2 got alter request for version 2
I0619 00:11:51.255270 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3b80a302aacb418091aee54ab4be7249" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:51.255631 25083 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
I0619 00:11:51.256501 25083 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 48 ms (attempt = 1)
W0619 00:11:51.256788 25083 catalog_manager.cc:3945] Failed to send ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
W0619 00:11:51.257560 25558 leader_election.cc:306] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.257863 25558 leader_election.cc:270] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.258400 25811 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.932s (election complete)
I0619 00:11:51.258626 25811 raft_consensus.cc:2487] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:51.259553 25080 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.260251 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
W0619 00:11:51.261843 25419 leader_election.cc:306] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.262094 25419 leader_election.cc:270] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.262871 25841 raft_consensus.cc:2732] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.909s (election complete)
I0619 00:11:51.263118 25841 raft_consensus.cc:2487] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:51.265040 25775 raft_consensus.cc:479] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.265964 25724 tablet_bootstrap.cc:578] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.265988 25775 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.266389 25775 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.266732 25325 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.267426 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:51.268376 25083 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.268386 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:51.268846 25624 raft_consensus.cc:2784] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.269085 25624 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.768s (vote granted)
I0619 00:11:51.269888 25727 tablet_bootstrap.cc:427] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:51.270476 25727 ts_tablet_manager.cc:1045] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.128s	user 0.006s	sys 0.001s
I0619 00:11:51.272545 25727 raft_consensus.cc:325] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.199378 25837 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:51.275183 25624 raft_consensus.cc:2245] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:51.279240 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.273968 25068 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 267 ms (attempt = 5)
W0619 00:11:51.280853 25068 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
W0619 00:11:51.281852 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.270397 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.273231 25727 raft_consensus.cc:351] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.282490 25727 raft_consensus.cc:628] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
W0619 00:11:51.283147 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.283164 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.273205 25741 ts_tablet_manager.cc:1382] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.283670 25741 log.cc:986] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/ac129ed44f68432f800420175aa0a630
I0619 00:11:51.284183 25741 ts_tablet_manager.cc:1403] T ac129ed44f68432f800420175aa0a630 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:51.283499 25727 consensus_queue.cc:227] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.284302 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_17ddf8eaaf154c45abbc00f1ed2c56bd"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:51.285228 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_17ddf8eaaf154c45abbc00f1ed2c56bd in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:51.287202 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times.
I0619 00:11:51.286310 25600 tablet_service.cc:800] Processing DeleteTablet for tablet aaf62da07a9740c1aa44359d3f604513 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.287835 25600 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.288192 25602 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.288276 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.288730 25600 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.288806 25598 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.288573 25844 tablet.cc:1195] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
I0619 00:11:51.288839 25601 tablet_service.cc:759] Processing CreateTablet for tablet 17e165e0995a4c029aa31392b9bbc04a (table=default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.289695 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 17e165e0995a4c029aa31392b9bbc04a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.292474 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:51.295964 25419 leader_election.cc:306] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:51.297049 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.297489 25423 leader_election.cc:379] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.299247 25083 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.299686 25846 tablet.cc:1195] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
I0619 00:11:51.300092 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.304311 25736 tablet_replica.cc:261] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:51.304574 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:51.304849 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:51.304814 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet ac129ed44f68432f800420175aa0a630 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) successfully deleted
I0619 00:11:51.305126 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(3b80a302aacb418091aee54ab4be7249)
W0619 00:11:51.305301 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:51.305371 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:51.305524 25067 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 145 ms (attempt = 4)
I0619 00:11:51.305796 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:51.306042 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:51.305979 25067 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 39 ms (attempt = 2)
W0619 00:11:51.306222 25067 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:51.306284 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(3b80a302aacb418091aee54ab4be7249)
W0619 00:11:51.306517 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet ac129ed44f68432f800420175aa0a630 because the tablet was not found. No further retry: Not found: Tablet not found: ac129ed44f68432f800420175aa0a630
I0619 00:11:51.306540 25736 raft_consensus.cc:2042] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.306896 25736 raft_consensus.cc:2069] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.307626 25423 leader_election.cc:270] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:51.308042 25725 tablet_bootstrap.cc:427] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.308393 25725 ts_tablet_manager.cc:1045] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.128s	user 0.002s	sys 0.003s
I0619 00:11:51.309172 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_ce167e3b89b8423a827ae8559f500233" } new_table_name: "default.table_24cb60c44bca436082a96bb044ee559b" alter_external_catalogs: true
I0619 00:11:51.312775 25844 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
W0619 00:11:51.314718 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.318408 25727 ts_tablet_manager.cc:1073] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.048s	user 0.004s	sys 0.001s
I0619 00:11:51.286934 25325 tablet_service.cc:759] Processing CreateTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 (table=default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:51.297147 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.299351 25819 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 2 got alter request for version 2
I0619 00:11:51.318970 25083 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.286756 25741 tablet_replica.cc:261] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:51.307981 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.320230 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.320452 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.278584 25730 ts_tablet_manager.cc:1382] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.320662 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.320803 25730 log.cc:986] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/03522c48c50943b38cae5aa88562e583
I0619 00:11:51.320855 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.321080 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.321275 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.321419 25730 ts_tablet_manager.cc:1403] T 03522c48c50943b38cae5aa88562e583 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:51.321482 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.321707 25741 raft_consensus.cc:2042] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.316184 25847 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.905s (election complete)
I0619 00:11:51.321964 25741 raft_consensus.cc:2069] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.321972 25847 raft_consensus.cc:2539] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:51.304447 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.323925 25847 raft_consensus.cc:593] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:51.324760 25087 catalog_manager.cc:3663] Got report from deleted tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.324374 25847 consensus_queue.cc:204] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.325405 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.325429 25725 raft_consensus.cc:325] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.326076 25725 raft_consensus.cc:351] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.326254 25725 raft_consensus.cc:628] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:51.328761 25725 consensus_queue.cc:227] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.330116 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 86959e7dda374ba9ac23d3c1fd30d993. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.330235 25725 ts_tablet_manager.cc:1073] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.000s
I0619 00:11:51.330519 25725 ts_tablet_manager.cc:1028] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:51.331252 25727 ts_tablet_manager.cc:1028] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:51.331457 25419 consensus_peers.cc:434] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 72e28bc6b57442b782d8aa07e1152cf4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:51.331090 25725 tablet_bootstrap.cc:427] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:51.329906 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 275 ms (attempt = 5)
W0619 00:11:51.331962 25067 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.331964 25727 tablet_bootstrap.cc:427] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:51.333431 25087 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.334882 25601 ts_tablet_manager.cc:1173] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.335176 25081 catalog_manager.cc:3663] Got report from deleted tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.336205 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.338325 25462 tablet_service.cc:800] Processing DeleteTablet for tablet aaf62da07a9740c1aa44359d3f604513 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:51.339020 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.343111 25837 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
I0619 00:11:51.344952 25741 ts_tablet_manager.cc:1369] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.345973 25727 tablet_bootstrap.cc:578] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:51.346724 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 1071 ms (attempt = 7)
W0619 00:11:51.350046 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:51.350148 25067 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 83 ms (attempt = 3)
W0619 00:11:51.350414 25067 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:51.350684 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.350966 25080 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_dd9523cf3fda45cd8bc544abb9dfba01" } new_table_name: "default.table_2c7f2c73082443ab869e2910097fa796" alter_external_catalogs: true
I0619 00:11:51.351660 25730 tablet_replica.cc:261] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:51.351928 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.352128 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(53688f0e4a604bc4a07dec9278a8dad6)
W0619 00:11:51.352146 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.352319 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.352515 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.352754 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.352955 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.353135 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.353308 25730 raft_consensus.cc:2042] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.357264 25811 raft_consensus.cc:457] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.357496 25811 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.658s (starting election)
I0619 00:11:51.357764 25811 raft_consensus.cc:479] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.358741 25811 leader_election.cc:243] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.359114 25811 leader_election.cc:243] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.361582 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acc89522b693498c987ca5d8ef8275ea" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:51.362957 25553 leader_election.cc:306] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.365226 25323 tablet_service.cc:759] Processing CreateTablet for tablet d62715e230154e2abe126ef97feec7ce (table=default.table_ce167e3b89b8423a827ae8559f500233 [id=fa287205f19a461fa0ba005f9991513b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.365309 25082 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 54 ms (attempt = 1)
W0619 00:11:51.365581 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:51.365819 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d62715e230154e2abe126ef97feec7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.373448 25080 catalog_manager.cc:2891] Scheduling retry of 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC with a delay of 55 ms (attempt = 1)
W0619 00:11:51.373852 25080 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a88e394b199431e8b794a1b1aec847f consensus state has no leader
I0619 00:11:51.374538 25323 ts_tablet_manager.cc:1173] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.374517 25068 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 276 ms (attempt = 5)
I0619 00:11:51.374709 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
W0619 00:11:51.374935 25068 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.375201 25724 tablet_bootstrap.cc:427] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
W0619 00:11:51.375509 25068 catalog_manager.cc:2910] Async tablet task aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet aaf62da07a9740c1aa44359d3f604513 consensus state has no leader
I0619 00:11:51.375587 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.375882 25724 ts_tablet_manager.cc:1045] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.129s	user 0.006s	sys 0.002s
I0619 00:11:51.379722 25601 tablet_service.cc:759] Processing CreateTablet for tablet 54adb588955b45b79de9752f254728ea (table=default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.380300 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 54adb588955b45b79de9752f254728ea. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.345454 25736 ts_tablet_manager.cc:1369] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.349905 25725 tablet_bootstrap.cc:578] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:51.354436 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.357400 25463 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:51.357496 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.358104 25468 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.358892 25464 tablet_service.cc:800] Processing DeleteTablet for tablet aaf62da07a9740c1aa44359d3f604513 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.359145 25470 tablet_service.cc:759] Processing CreateTablet for tablet 17e165e0995a4c029aa31392b9bbc04a (table=default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.360644 25467 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.361438 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acc89522b693498c987ca5d8ef8275ea" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:51.385702 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 17e165e0995a4c029aa31392b9bbc04a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.385797 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.386262 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.391487 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.353570 25730 raft_consensus.cc:2069] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.392884 25730 ts_tablet_manager.cc:1369] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.393473 25819 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 2 got alter request for version 2
I0619 00:11:51.394142 25323 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.396030 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.397117 25601 ts_tablet_manager.cc:1173] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.399024 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) successfully deleted
I0619 00:11:51.399742 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.400025 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.400437 25325 ts_tablet_manager.cc:1173] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.400902 25727 tablet_bootstrap.cc:427] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:51.401235 25727 ts_tablet_manager.cc:1045] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.070s	user 0.005s	sys 0.000s
I0619 00:11:51.402467 25727 raft_consensus.cc:325] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.403111 25727 raft_consensus.cc:351] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.403312 25727 raft_consensus.cc:628] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:51.404196 25467 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:51.405216 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.407598 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 397ae10bb6d841b4ba561ec431d29864 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:51.409297 25796 raft_consensus.cc:457] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:51.409500 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.409545 25796 raft_consensus.cc:2732] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.140s (starting election)
I0619 00:11:51.409893 25796 raft_consensus.cc:479] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.410861 25796 leader_election.cc:243] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.411238 25796 leader_election.cc:243] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.379287 25724 raft_consensus.cc:325] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.411902 25724 raft_consensus.cc:351] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.412130 25724 raft_consensus.cc:628] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:51.411702 25844 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
I0619 00:11:51.414217 25470 ts_tablet_manager.cc:1173] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.414710 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "53688f0e4a604bc4a07dec9278a8dad6" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:51.415184 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.415315 25558 leader_election.cc:306] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.403838 25727 consensus_queue.cc:227] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.415650 25558 leader_election.cc:270] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.416116 25811 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.763s (election complete)
I0619 00:11:51.416285 25837 tablet.cc:1195] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
I0619 00:11:51.416342 25811 raft_consensus.cc:2487] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:51.417033 25419 leader_election.cc:306] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.420306 25811 raft_consensus.cc:457] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.420526 25811 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.920s (starting election)
I0619 00:11:51.421010 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.420790 25811 raft_consensus.cc:479] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:51.421524 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.421774 25811 leader_election.cc:243] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.422155 25811 leader_election.cc:243] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.425110 25844 tablet.cc:1195] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
I0619 00:11:51.425392 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee4c4769f75141f9b2ef864ab8a8d9e0" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:51.427235 25553 leader_election.cc:306] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.428462 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "53688f0e4a604bc4a07dec9278a8dad6" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:51.429996 25069 catalog_manager.cc:2891] Scheduling retry of 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC with a delay of 67 ms (attempt = 2)
W0619 00:11:51.430271 25069 catalog_manager.cc:2910] Async tablet task 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a88e394b199431e8b794a1b1aec847f consensus state has no leader
W0619 00:11:51.431181 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.432494 25724 consensus_queue.cc:227] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.433715 25080 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.434418 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
W0619 00:11:51.434842 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.435317 25325 tablet_service.cc:800] Processing DeleteTablet for tablet aaf62da07a9740c1aa44359d3f604513 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.436393 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 74 ms (attempt = 2)
W0619 00:11:51.436662 25067 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:51.437029 25067 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 158 ms (attempt = 4)
W0619 00:11:51.437265 25067 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:51.437558 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) successfully deleted
W0619 00:11:51.437861 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.439379 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet acc89522b693498c987ca5d8ef8275ea from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:51.438315 25724 ts_tablet_manager.cc:1073] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.062s	user 0.004s	sys 0.000s
I0619 00:11:51.439379 25083 catalog_manager.cc:3786] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:51.440379 25724 ts_tablet_manager.cc:1028] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:51.440497 25083 catalog_manager.cc:3663] Got report from deleted tablet 53688f0e4a604bc4a07dec9278a8dad6 (table default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:51.440451 25470 tablet_service.cc:759] Processing CreateTablet for tablet 54adb588955b45b79de9752f254728ea (table=default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.440836 25724 tablet_bootstrap.cc:427] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:51.442152 25082 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.415392 25601 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.442828 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.441424 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 54adb588955b45b79de9752f254728ea. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:51.444454 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet f372b9fb174844eca1662c62039a3c71 because the tablet was not found. No further retry: Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
I0619 00:11:51.445612 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.445844 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.446467 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.446492 25601 tablet_service.cc:759] Processing CreateTablet for tablet a284677c4b7f46e18e25edee6024e3eb (table=default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.447136 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a284677c4b7f46e18e25edee6024e3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.430135 25727 ts_tablet_manager.cc:1073] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.029s	user 0.004s	sys 0.000s
W0619 00:11:51.445580 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e because the tablet was not found. No further retry: Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
I0619 00:11:51.449447 25725 tablet_bootstrap.cc:427] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.450067 25725 ts_tablet_manager.cc:1045] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.119s	user 0.004s	sys 0.002s
I0619 00:11:51.450531 25727 ts_tablet_manager.cc:1028] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:51.451092 25727 tablet_bootstrap.cc:427] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:51.451771 25807 raft_consensus.cc:457] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.452026 25807 raft_consensus.cc:2732] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.246s (starting election)
I0619 00:11:51.452265 25807 raft_consensus.cc:479] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.452158 25725 raft_consensus.cc:325] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.453091 25725 raft_consensus.cc:351] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.453253 25807 leader_election.cc:243] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.453364 25725 raft_consensus.cc:628] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:51.453649 25807 leader_election.cc:243] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.453946 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.454450 25807 raft_consensus.cc:457] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.454685 25807 raft_consensus.cc:2732] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.736s (starting election)
I0619 00:11:51.454095 25725 consensus_queue.cc:227] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.454923 25807 raft_consensus.cc:479] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.455873 25807 leader_election.cc:243] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.456238 25807 leader_election.cc:243] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.458631 25724 tablet_bootstrap.cc:578] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.458993 25725 ts_tablet_manager.cc:1073] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I0619 00:11:51.459530 25725 ts_tablet_manager.cc:1028] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:51.460224 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 53688f0e4a604bc4a07dec9278a8dad6 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.461372 25325 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.461611 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.466152 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee4c4769f75141f9b2ef864ab8a8d9e0" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:51.467519 25467 tablet_service.cc:759] Processing CreateTablet for tablet a284677c4b7f46e18e25edee6024e3eb (table=default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.468137 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a284677c4b7f46e18e25edee6024e3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.469115 25325 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.470053 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.470654 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.472266 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet acc89522b693498c987ca5d8ef8275ea
I0619 00:11:51.472649 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acc89522b693498c987ca5d8ef8275ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:51.473065 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acc89522b693498c987ca5d8ef8275ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:51.473441 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acc89522b693498c987ca5d8ef8275ea on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:51.473840 25080 catalog_manager.cc:4596] ReplaceTablet: tablet acc89522b693498c987ca5d8ef8275ea deleted and replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:51.450022 25741 ts_tablet_manager.cc:1382] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I0619 00:11:51.475704 25741 log.cc:986] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:51.476555 25727 tablet_bootstrap.cc:578] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:51.480293 25741 ts_tablet_manager.cc:1403] T 72e28bc6b57442b782d8aa07e1152cf4 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:51.481190 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:51.482550 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:51.483196 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.483186 25558 leader_election.cc:306] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.483464 25558 leader_election.cc:270] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.484138 25811 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.031s (election complete)
I0619 00:11:51.484352 25811 raft_consensus.cc:2487] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:51.483775 25423 leader_election.cc:306] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:11:51.485481 25423 leader_election.cc:270] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.485981 25796 raft_consensus.cc:2732] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.620s (election complete)
I0619 00:11:51.486209 25796 raft_consensus.cc:2487] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:51.486510 25837 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
W0619 00:11:51.488075 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7f60ea5c5def47ee846d0186bbc90c18 already in progress: creating tablet
I0619 00:11:51.488517 25071 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 58 ms (attempt = 1)
W0619 00:11:51.489307 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:51.490264 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 36 ms (attempt = 2)
I0619 00:11:51.495021 25736 ts_tablet_manager.cc:1382] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.495417 25736 log.cc:986] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/3b80a302aacb418091aee54ab4be7249
I0619 00:11:51.496134 25736 ts_tablet_manager.cc:1403] T 3b80a302aacb418091aee54ab4be7249 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:51.498538 25069 catalog_manager.cc:2891] Scheduling retry of 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC with a delay of 97 ms (attempt = 3)
I0619 00:11:51.498637 25741 tablet_replica.cc:261] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
W0619 00:11:51.498831 25069 catalog_manager.cc:2910] Async tablet task 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a88e394b199431e8b794a1b1aec847f consensus state has no leader
I0619 00:11:51.499001 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:51.499424 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:51.499765 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:51.502674 25083 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.503723 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.503888 25325 tablet_service.cc:759] Processing CreateTablet for tablet 17e165e0995a4c029aa31392b9bbc04a (table=default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.504468 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 17e165e0995a4c029aa31392b9bbc04a. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:51.505452 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.506127 25598 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.506718 25600 tablet_service.cc:759] Processing CreateTablet for tablet 19ad911ec6704aef85d9b7f172f08bdc (table=default.table_17ddf8eaaf154c45abbc00f1ed2c56bd [id=9306f61885914c84acf74557c89d2c81]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.506892 25598 tablet_service.cc:800] Processing DeleteTablet for tablet acc89522b693498c987ca5d8ef8275ea with delete_type TABLET_DATA_DELETED (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.506940 25725 tablet_bootstrap.cc:427] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:51.507335 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 19ad911ec6704aef85d9b7f172f08bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.509465 25601 ts_tablet_manager.cc:1173] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.509727 25725 tablet_bootstrap.cc:578] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:51.510756 25837 tablet.cc:1195] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
I0619 00:11:51.511811 25083 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_e248c37d574c4073ad57f7b09e7bf793" }
I0619 00:11:51.514171 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 71 ms (attempt = 3)
W0619 00:11:51.514449 25067 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
W0619 00:11:51.516098 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.516816 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet c18355df78d344839a56189c7549ab1f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c18355df78d344839a56189c7549ab1f already in progress: creating tablet
I0619 00:11:51.517078 25071 catalog_manager.cc:2891] Scheduling retry of c18355df78d344839a56189c7549ab1f Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 59 ms (attempt = 1)
I0619 00:11:51.517202 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.517946 25290 leader_election.cc:306] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: fc22ad98ed2e4e3eb2c62360626a6f87
I0619 00:11:51.518245 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3b5e020c7a0148b48bc211eb1dc414ec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:51.518792 25290 leader_election.cc:306] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: fc22ad98ed2e4e3eb2c62360626a6f87
W0619 00:11:51.519117 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3b5e020c7a0148b48bc211eb1dc414ec in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:51.519125 25290 leader_election.cc:270] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.519742 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_33483e8524674ab0a5207eb14818073b" }
I0619 00:11:51.520208 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) successfully deleted
I0619 00:11:51.520462 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:51.521114 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 1d10fb4561924e2e8d53c7a0b22a426e (table default.table_f4fd4524d920428fa515c1266d90fb2a [id=b46718e25afb4950af6cacec1d51a9ba]) successfully deleted
I0619 00:11:51.521816 25807 raft_consensus.cc:2732] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.223s (election complete)
I0619 00:11:51.522040 25807 raft_consensus.cc:2487] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:51.523191 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.523147 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:51.525068 25323 tablet_service.cc:800] Processing DeleteTablet for tablet fc22ad98ed2e4e3eb2c62360626a6f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.527412 25464 tablet_service.cc:759] Processing CreateTablet for tablet 19ad911ec6704aef85d9b7f172f08bdc (table=default.table_17ddf8eaaf154c45abbc00f1ed2c56bd [id=9306f61885914c84acf74557c89d2c81]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.529136 25844 tablet.cc:1195] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
I0619 00:11:51.531406 25736 tablet_replica.cc:261] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:51.532071 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:51.532161 25082 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.532366 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.533870 25467 ts_tablet_manager.cc:1173] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:51.534592 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.518232 25724 tablet_bootstrap.cc:427] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:51.535812 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.536036 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.536178 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.534857 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.536257 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.536548 25724 ts_tablet_manager.cc:1045] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.096s	user 0.004s	sys 0.001s
I0619 00:11:51.536607 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.536864 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.537073 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.537170 25600 ts_tablet_manager.cc:1173] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.537276 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.537155 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.537458 25736 raft_consensus.cc:2042] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:51.537674 25736 consensus_peers.cc:456] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:51.537722 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.537919 25736 consensus_peers.cc:456] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:51.538318 25736 pending_rounds.cc:63] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Trying to abort 1 pending transactions.
I0619 00:11:51.538215 25325 ts_tablet_manager.cc:1173] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.538480 25736 pending_rounds.cc:68] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Aborting transaction as it isn't in flight: id { term: 1 index: 6 } timestamp: 6264390907951005696 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "72e28bc6b57442b782d8aa07e1152cf4" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } schema_version: 2 new_table_name: "default.table_ce167e3b89b8423a827ae8559f500233" dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" }
W0619 00:11:51.539047 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
I0619 00:11:51.539393 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 32 ms (attempt = 2)
I0619 00:11:51.539072 25724 raft_consensus.cc:325] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.539703 25724 raft_consensus.cc:351] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.539788 25727 tablet_bootstrap.cc:427] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:51.539894 25724 raft_consensus.cc:628] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:51.539830 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 1d10fb4561924e2e8d53c7a0b22a426e because the tablet was not found. No further retry: Not found: Tablet not found: 1d10fb4561924e2e8d53c7a0b22a426e
I0619 00:11:51.540104 25727 ts_tablet_manager.cc:1045] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.089s	user 0.005s	sys 0.000s
I0619 00:11:51.540802 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:51.540401 25724 consensus_queue.cc:227] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:51.541260 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 03522c48c50943b38cae5aa88562e583 because the tablet was not found. No further retry: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
W0619 00:11:51.541715 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:51.542057 25071 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 32 ms (attempt = 2)
W0619 00:11:51.542551 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.543575 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.544263 25467 tablet_service.cc:800] Processing DeleteTablet for tablet acc89522b693498c987ca5d8ef8275ea with delete_type TABLET_DATA_DELETED (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.544649 25083 catalog_manager.cc:3663] Got report from deleted tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293): Sending delete request for this tablet
W0619 00:11:51.544888 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.545064 25724 ts_tablet_manager.cc:1073] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I0619 00:11:51.545437 25724 ts_tablet_manager.cc:1028] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:51.545436 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:51.545831 25724 tablet_bootstrap.cc:427] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:51.546440 25727 raft_consensus.cc:325] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.547075 25727 raft_consensus.cc:351] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.509713 25730 ts_tablet_manager.cc:1382] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.547253 25727 raft_consensus.cc:628] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:51.547361 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.547492 25730 log.cc:986] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/53688f0e4a604bc4a07dec9278a8dad6
I0619 00:11:51.547685 25727 consensus_queue.cc:227] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.548545 25730 ts_tablet_manager.cc:1403] T 53688f0e4a604bc4a07dec9278a8dad6 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:51.549618 25068 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 533 ms (attempt = 6)
W0619 00:11:51.549907 25068 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.550276 25470 ts_tablet_manager.cc:1173] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.551908 25730 tablet_replica.cc:261] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:51.552026 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 19ad911ec6704aef85d9b7f172f08bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.552137 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:51.552335 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:51.552525 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:51.552753 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:51.552721 25083 catalog_manager.cc:3663] Got report from deleted tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293): Sending delete request for this tablet
I0619 00:11:51.552978 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:51.553179 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:51.553375 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:51.553356 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:51.555663 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) successfully deleted
W0619 00:11:51.556023 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 1d10fb4561924e2e8d53c7a0b22a426e because the tablet was not found. No further retry: Not found: Tablet not found: 1d10fb4561924e2e8d53c7a0b22a426e
W0619 00:11:51.556721 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:51.556721 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.556944 25067 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 291 ms (attempt = 5)
I0619 00:11:51.524628 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:51.558503 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.560988 25600 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.561393 25601 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.562708 25467 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:51.563058 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.563122 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.563349 25464 ts_tablet_manager.cc:1173] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:51.563913 25290 leader_election.cc:306] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
W0619 00:11:51.564417 25290 leader_election.cc:306] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
I0619 00:11:51.564743 25290 leader_election.cc:270] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.565192 25807 raft_consensus.cc:2732] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.681s (election complete)
I0619 00:11:51.565402 25807 raft_consensus.cc:2487] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:51.566174 25325 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.539593 25736 raft_consensus.cc:2069] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.568661 25725 tablet_bootstrap.cc:427] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.568579 25748 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:37587,127.24.119.61:46325,127.24.119.60:57058
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:46325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.119.61
--webserver_port=33317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:11:51.568994 25725 ts_tablet_manager.cc:1045] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.064s	user 0.004s	sys 0.001s
I0619 00:11:51.570920 25725 raft_consensus.cc:325] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.571364 25724 tablet_bootstrap.cc:578] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.571559 25725 raft_consensus.cc:351] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.571805 25725 raft_consensus.cc:628] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:51.572254 25725 consensus_queue.cc:227] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:51.573818 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f372b9fb174844eca1662c62039a3c71 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f372b9fb174844eca1662c62039a3c71 already in progress: creating tablet
I0619 00:11:51.574544 25067 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 294 ms (attempt = 5)
W0619 00:11:51.574748 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.575489 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet ac129ed44f68432f800420175aa0a630 because the tablet was not found. No further retry: Not found: Tablet not found: ac129ed44f68432f800420175aa0a630
I0619 00:11:51.579332 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 775f0b6b309c49278066a5e7cf623af2
I0619 00:11:51.579659 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.580030 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.580391 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.553548 25730 raft_consensus.cc:2042] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.581642 25727 ts_tablet_manager.cc:1073] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.041s	user 0.005s	sys 0.000s
I0619 00:11:51.581938 25730 raft_consensus.cc:2069] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.582010 25727 ts_tablet_manager.cc:1028] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:51.578717 25083 catalog_manager.cc:3663] Got report from deleted tablet 53688f0e4a604bc4a07dec9278a8dad6 (table default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:51.582368 25727 tablet_bootstrap.cc:427] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:51.583236 25730 ts_tablet_manager.cc:1369] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.582373 25725 ts_tablet_manager.cc:1073] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I0619 00:11:51.584223 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.585114 25727 tablet_bootstrap.cc:578] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:51.584878 25725 ts_tablet_manager.cc:1028] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:51.584995 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 53688f0e4a604bc4a07dec9278a8dad6 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.586416 25725 tablet_bootstrap.cc:427] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:51.586110 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.587004 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(fc22ad98ed2e4e3eb2c62360626a6f87)
I0619 00:11:51.587229 25741 raft_consensus.cc:2042] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.587489 25741 raft_consensus.cc:2069] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.588665 25741 ts_tablet_manager.cc:1369] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.589295 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.590531 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:51.591958 25736 ts_tablet_manager.cc:1369] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.592957 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 53688f0e4a604bc4a07dec9278a8dad6 (table default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]) successfully deleted
I0619 00:11:51.596437 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.596982 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.597149 25069 catalog_manager.cc:2891] Scheduling retry of 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC with a delay of 148 ms (attempt = 4)
W0619 00:11:51.597405 25069 catalog_manager.cc:2910] Async tablet task 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a88e394b199431e8b794a1b1aec847f consensus state has no leader
I0619 00:11:51.598197 25819 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 2 got alter request for version 2
I0619 00:11:51.601634 25725 tablet_bootstrap.cc:578] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:51.603221 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.603878 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.604169 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.604692 25601 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.604964 25080 catalog_manager.cc:3663] Got report from deleted tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) (replaced by tablet d62715e230154e2abe126ef97feec7ce): Sending delete request for this tablet
I0619 00:11:51.605280 25080 catalog_manager.cc:3663] Got report from deleted tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (replaced by tablet a85577b0beba4276b963e5c360544bc7): Sending delete request for this tablet
I0619 00:11:51.605427 25464 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.606181 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 72e28bc6b57442b782d8aa07e1152cf4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d62715e230154e2abe126ef97feec7ce)
I0619 00:11:51.606544 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.607718 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 168 ms (attempt = 4)
W0619 00:11:51.607982 25067 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:51.608306 25067 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 296 ms (attempt = 5)
I0619 00:11:51.608255 25601 tablet_service.cc:759] Processing CreateTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc (table=default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:51.608551 25067 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:51.608916 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c0614e9ed96420886243a6868f2e75e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.609877 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 546 ms (attempt = 6)
W0619 00:11:51.610213 25067 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.611011 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 981d862d1bab4b14bb10e6fc2174e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.613210 25323 tablet_service.cc:759] Processing CreateTablet for tablet 54adb588955b45b79de9752f254728ea (table=default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.613785 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 54adb588955b45b79de9752f254728ea. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.617923 25796 raft_consensus.cc:457] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.618197 25796 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.683s (starting election)
I0619 00:11:51.618453 25600 tablet_service.cc:759] Processing CreateTablet for tablet 54373917567a479d93dc8a3fd006d2fe (table=default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.619068 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 54373917567a479d93dc8a3fd006d2fe. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:51.620985 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.622140 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.622524 25844 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
I0619 00:11:51.618476 25796 raft_consensus.cc:479] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.623798 25837 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
W0619 00:11:51.625136 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.625242 25796 leader_election.cc:243] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.626147 25796 leader_election.cc:243] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.627818 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "70b9c8c4af0945b69b5577e47b61c054" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:51.629465 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.630405 25419 leader_election.cc:306] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:51.630568 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.631281 25423 leader_election.cc:306] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054
I0619 00:11:51.631554 25423 leader_election.cc:270] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.631906 25323 ts_tablet_manager.cc:1173] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.632277 25796 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.957s (election complete)
I0619 00:11:51.632520 25796 raft_consensus.cc:2487] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:51.636139 25600 ts_tablet_manager.cc:1173] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.637576 25464 tablet_service.cc:759] Processing CreateTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc (table=default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.638170 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 981d862d1bab4b14bb10e6fc2174e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.638294 25730 ts_tablet_manager.cc:1382] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.638520 25730 log.cc:986] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:51.638981 25730 ts_tablet_manager.cc:1403] T cb57d196f0724247b6a7f9a34be8377a P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:51.639411 25741 ts_tablet_manager.cc:1382] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.641419 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.641894 25748 master_main.cc:80] Initializing master server...
I0619 00:11:51.643043 25727 tablet_bootstrap.cc:427] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:51.643373 25727 ts_tablet_manager.cc:1045] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.061s	user 0.004s	sys 0.002s
I0619 00:11:51.644665 25727 raft_consensus.cc:325] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.645296 25727 raft_consensus.cc:351] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.645495 25727 raft_consensus.cc:628] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:51.645965 25727 consensus_queue.cc:227] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.648012 25464 ts_tablet_manager.cc:1173] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.649266 25727 ts_tablet_manager.cc:1073] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I0619 00:11:51.650856 25080 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_17ddf8eaaf154c45abbc00f1ed2c56bd" } new_table_name: "default.table_5a3f045aae58498688f23e73d119682e" alter_external_catalogs: true
I0619 00:11:51.651348 25727 ts_tablet_manager.cc:1028] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:51.651562 25068 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 559 ms (attempt = 6)
I0619 00:11:51.651756 25727 tablet_bootstrap.cc:427] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:51.651875 25068 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.657935 25601 ts_tablet_manager.cc:1173] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.658923 25748 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 443913us
I0619 00:11:51.658943 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c8d425c090594b149578c26728881c9d"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:51.659474 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.659811 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c8d425c090594b149578c26728881c9d in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:51.661607 25080 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 41 ms (attempt = 1)
W0619 00:11:51.661906 25080 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:51.662475 25325 tablet_service.cc:759] Processing CreateTablet for tablet 19ad911ec6704aef85d9b7f172f08bdc (table=default.table_17ddf8eaaf154c45abbc00f1ed2c56bd [id=9306f61885914c84acf74557c89d2c81]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.662534 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) successfully deleted
I0619 00:11:51.663105 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 19ad911ec6704aef85d9b7f172f08bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.663040 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 292b164659e343158de8e7e3eeb9be31 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) successfully deleted
I0619 00:11:51.664988 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 1035 ms (attempt = 7)
W0619 00:11:51.665263 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.665951 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_1563f330df024855ab3d01187bdc7661"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:51.666828 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1563f330df024855ab3d01187bdc7661 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:51.648833 25470 tablet_service.cc:759] Processing CreateTablet for tablet 54373917567a479d93dc8a3fd006d2fe (table=default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.649734 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 72e28bc6b57442b782d8aa07e1152cf4 with delete_type TABLET_DATA_DELETED (replaced by tablet d62715e230154e2abe126ef97feec7ce) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.650277 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.654121 25796 raft_consensus.cc:457] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.668305 25796 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.721s (starting election)
I0619 00:11:51.668570 25796 raft_consensus.cc:479] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.670420 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 54373917567a479d93dc8a3fd006d2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.671069 25796 leader_election.cc:243] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.672055 25796 leader_election.cc:243] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.650936 25736 ts_tablet_manager.cc:1382] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I0619 00:11:51.672811 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aaf62da07a9740c1aa44359d3f604513" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:51.673260 25354 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.834s (vote granted)
I0619 00:11:51.673471 25354 raft_consensus.cc:2245] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:51.673997 25736 log.cc:986] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:51.674559 25419 leader_election.cc:379] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.675148 25419 leader_election.cc:270] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:51.675376 25323 tablet_service.cc:759] Processing CreateTablet for tablet a284677c4b7f46e18e25edee6024e3eb (table=default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.675834 25796 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.629s (election complete)
I0619 00:11:51.675804 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aaf62da07a9740c1aa44359d3f604513" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:51.676082 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a284677c4b7f46e18e25edee6024e3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.676095 25796 raft_consensus.cc:2539] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:51.676259 25624 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.107s (vote granted)
I0619 00:11:51.676393 25796 raft_consensus.cc:457] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:51.676483 25624 raft_consensus.cc:2245] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:51.676662 25796 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.759s (starting election)
I0619 00:11:51.676960 25796 raft_consensus.cc:2784] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.678678 25724 tablet_bootstrap.cc:427] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:51.679219 25724 ts_tablet_manager.cc:1045] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.133s	user 0.005s	sys 0.005s
I0619 00:11:51.679677 25741 log.cc:986] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/fc22ad98ed2e4e3eb2c62360626a6f87
I0619 00:11:51.680222 25741 ts_tablet_manager.cc:1403] T fc22ad98ed2e4e3eb2c62360626a6f87 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:51.681077 25423 leader_election.cc:379] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.681982 25736 ts_tablet_manager.cc:1403] T 72e28bc6b57442b782d8aa07e1152cf4 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:51.682034 25724 raft_consensus.cc:325] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.683066 25724 raft_consensus.cc:351] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.683357 25724 raft_consensus.cc:628] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:51.684036 25724 consensus_queue.cc:227] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.684615 25796 raft_consensus.cc:479] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.685577 25796 leader_election.cc:243] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.686751 25796 leader_election.cc:243] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:51.687378 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.687408 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aaf62da07a9740c1aa44359d3f604513" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:51.687862 25354 raft_consensus.cc:2784] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.688091 25354 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.637s (vote granted)
W0619 00:11:51.688199 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7f60ea5c5def47ee846d0186bbc90c18 already in progress: creating tablet
I0619 00:11:51.688452 25071 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 60 ms (attempt = 1)
I0619 00:11:51.689761 25081 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_de92feb7880c4ea2b3fd76c65c6b3df5" } new_table_name: "default.table_e4aaded22fbb4c60a591e5ff45bfe2bc" alter_external_catalogs: true
W0619 00:11:51.690189 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.691730 25724 ts_tablet_manager.cc:1073] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I0619 00:11:51.692858 25325 ts_tablet_manager.cc:1173] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.693296 25354 raft_consensus.cc:2245] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:51.694608 25419 leader_election.cc:379] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.694900 25419 leader_election.cc:270] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
W0619 00:11:51.695008 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e because the tablet was not found. No further retry: Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
I0619 00:11:51.695449 25847 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.835s (election complete)
I0619 00:11:51.695732 25847 raft_consensus.cc:2539] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:51.696065 25847 raft_consensus.cc:593] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:51.694780 25725 tablet_bootstrap.cc:427] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.696491 25847 consensus_queue.cc:204] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.697129 25725 ts_tablet_manager.cc:1045] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.111s	user 0.006s	sys 0.003s
I0619 00:11:51.699043 25601 tablet_service.cc:759] Processing CreateTablet for tablet c244dec62a5b4ee492a9a110b656a561 (table=default.table_3b5e020c7a0148b48bc211eb1dc414ec [id=9f88d7e327314448a85e60f6aa682490]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.699601 25741 tablet_replica.cc:261] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
W0619 00:11:51.699725 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.699779 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c244dec62a5b4ee492a9a110b656a561. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.700073 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.700577 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.700899 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.700258 25725 raft_consensus.cc:325] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.701175 25725 raft_consensus.cc:351] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.701274 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.701479 25725 raft_consensus.cc:628] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:51.701771 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.702145 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.702456 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.702747 25741 raft_consensus.cc:2042] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.702370 25725 consensus_queue.cc:227] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.703172 25741 raft_consensus.cc:2069] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:51.705274 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.703872 25464 tablet_service.cc:759] Processing CreateTablet for tablet c244dec62a5b4ee492a9a110b656a561 (table=default.table_3b5e020c7a0148b48bc211eb1dc414ec [id=9f88d7e327314448a85e60f6aa682490]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.705943 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet fc22ad98ed2e4e3eb2c62360626a6f87 (table default.table_9bbf1c0d460e4aafa3551e40c25513c4 [id=60acf6cc649c418c86934f212402a49d]) successfully deleted
I0619 00:11:51.706030 25725 ts_tablet_manager.cc:1073] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
W0619 00:11:51.706912 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
W0619 00:11:51.706919 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.706630 25725 ts_tablet_manager.cc:1028] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:51.707592 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c244dec62a5b4ee492a9a110b656a561. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.707870 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.708416 25727 tablet_bootstrap.cc:578] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:51.708811 25730 tablet_replica.cc:261] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:51.709256 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.709584 25748 fs_manager.cc:261] Metadata directory not provided
I0619 00:11:51.709636 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.709945 25748 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:11:51.710065 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.710553 25819 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 2 got alter request for version 2
I0619 00:11:51.711812 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.709606 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 74 ms (attempt = 2)
I0619 00:11:51.712241 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.712631 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.712967 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:51.713884 25087 catalog_manager.cc:3663] Got report from deleted tablet aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:51.714102 25082 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:51.714596 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aaf62da07a9740c1aa44359d3f604513 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.714850 25082 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 46 ms (attempt = 1)
W0619 00:11:51.715111 25082 catalog_manager.cc:3945] Failed to send 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.692360 25724 ts_tablet_manager.cc:1028] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
W0619 00:11:51.712993 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:51.719852 25325 tablet_service.cc:800] Processing DeleteTablet for tablet acc89522b693498c987ca5d8ef8275ea with delete_type TABLET_DATA_DELETED (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.713304 25470 ts_tablet_manager.cc:1173] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:51.727393 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.729913 25844 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
W0619 00:11:51.732218 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.729127 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:51.733592 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet 53688f0e4a604bc4a07dec9278a8dad6 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:51.733642 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:51.733849 25087 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 36 ms (attempt = 1)
W0619 00:11:51.734163 25087 catalog_manager.cc:3945] Failed to send 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.729535 25068 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 1037 ms (attempt = 7)
W0619 00:11:51.734742 25068 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:51.735525 25736 tablet_replica.cc:261] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:51.735831 25837 tablet.cc:1195] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
I0619 00:11:51.736953 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:51.737556 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:51.737804 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:51.738029 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:51.733875 25071 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 84 ms (attempt = 3)
I0619 00:11:51.738279 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:51.738499 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:51.738364 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aaf62da07a9740c1aa44359d3f604513" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:51.738768 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:51.738835 25624 raft_consensus.cc:2784] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.739085 25624 raft_consensus.cc:2732] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.039s (vote granted)
I0619 00:11:51.707592 25725 tablet_bootstrap.cc:427] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:51.740715 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.738032 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.742264 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ab760159a5834462a648c3167c0456d9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:51.733136 25724 tablet_bootstrap.cc:427] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:51.743166 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ab760159a5834462a648c3167c0456d9 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:51.742282 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 53688f0e4a604bc4a07dec9278a8dad6 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 53688f0e4a604bc4a07dec9278a8dad6 already in progress: creating tablet
I0619 00:11:51.743434 25624 raft_consensus.cc:2245] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:51.744148 25071 catalog_manager.cc:2891] Scheduling retry of 53688f0e4a604bc4a07dec9278a8dad6 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 54 ms (attempt = 1)
I0619 00:11:51.744755 25423 leader_election.cc:379] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.744814 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.745806 25069 catalog_manager.cc:2891] Scheduling retry of 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC with a delay of 298 ms (attempt = 5)
W0619 00:11:51.746115 25069 catalog_manager.cc:2910] Async tablet task 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a88e394b199431e8b794a1b1aec847f consensus state has no leader
I0619 00:11:51.746284 25724 tablet_bootstrap.cc:578] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.746630 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.713274 25730 raft_consensus.cc:2042] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.747479 25325 tablet_service.cc:759] Processing CreateTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc (table=default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.747562 25082 master_service.cc:523] ReplaceTablet: received request to replace tablet 7f60ea5c5def47ee846d0186bbc90c18 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:51.747900 25811 raft_consensus.cc:576] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I0619 00:11:51.748072 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 981d862d1bab4b14bb10e6fc2174e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.748212 25730 raft_consensus.cc:2069] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.749886 25730 ts_tablet_manager.cc:1369] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.755506 25736 raft_consensus.cc:2042] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:51.757601 25736 consensus_peers.cc:456] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:51.757889 25736 consensus_peers.cc:456] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:51.758280 25736 pending_rounds.cc:63] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Trying to abort 1 pending transactions.
I0619 00:11:51.758463 25736 pending_rounds.cc:68] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390908717498368 op_type: NO_OP noop_request { }
I0619 00:11:51.758970 25736 raft_consensus.cc:2622] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:51.759238 25736 raft_consensus.cc:2069] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:51.759755 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 because the tablet was not found. No further retry: Not found: Tablet not found: 292b164659e343158de8e7e3eeb9be31
I0619 00:11:51.761390 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.762349 25080 catalog_manager.cc:3663] Got report from deleted tablet aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:51.764047 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aaf62da07a9740c1aa44359d3f604513 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.765980 25624 raft_consensus.cc:1081] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:51.778650 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 286 ms (attempt = 5)
I0619 00:11:51.770875 25069 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 74 ms (attempt = 2)
W0619 00:11:51.778931 25067 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:51.778946 25323 ts_tablet_manager.cc:1173] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:51.779080 25069 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.779955 25336 tablet_service.cc:759] Processing CreateTablet for tablet 54373917567a479d93dc8a3fd006d2fe (table=default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.780549 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 54373917567a479d93dc8a3fd006d2fe. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.780732 25724 tablet_bootstrap.cc:427] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:51.780848 25081 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3" }
I0619 00:11:51.781054 25724 ts_tablet_manager.cc:1045] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.065s	user 0.003s	sys 0.001s
I0619 00:11:51.782400 25724 raft_consensus.cc:325] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.783222 25724 raft_consensus.cc:351] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.783430 25724 raft_consensus.cc:628] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:51.783864 25741 ts_tablet_manager.cc:1369] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.783973 25724 consensus_queue.cc:227] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:51.787194 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.788031 25748 fs_manager.cc:397] Time spent opening directory manager: real 0.040s	user 0.002s	sys 0.002s
I0619 00:11:51.789657 25748 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
W0619 00:11:51.790833 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.791223 25748 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:11:51.798185 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:51.799008 25080 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 38 ms (attempt = 1)
I0619 00:11:51.799808 25326 tablet_service.cc:759] Processing CreateTablet for tablet c244dec62a5b4ee492a9a110b656a561 (table=default.table_3b5e020c7a0148b48bc211eb1dc414ec [id=9f88d7e327314448a85e60f6aa682490]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.800451 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c244dec62a5b4ee492a9a110b656a561. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.800572 25727 tablet_bootstrap.cc:427] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:51.800927 25727 ts_tablet_manager.cc:1045] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.149s	user 0.004s	sys 0.000s
I0619 00:11:51.802222 25727 raft_consensus.cc:325] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.802831 25727 raft_consensus.cc:351] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.802875 25330 tablet_service.cc:800] Processing DeleteTablet for tablet aaf62da07a9740c1aa44359d3f604513 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.804750 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 67 ms (attempt = 2)
W0619 00:11:51.805060 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.805392 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 53688f0e4a604bc4a07dec9278a8dad6 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.806277 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_14bc9661c33c47beab02a4d149b07c42"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:51.806738 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 4c0614e9ed96420886243a6868f2e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.806726 25080 catalog_manager.cc:3945] Failed to send 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
W0619 00:11:51.807173 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_14bc9661c33c47beab02a4d149b07c42 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:51.808810 25807 raft_consensus.cc:457] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.809042 25807 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.920s (starting election)
I0619 00:11:51.809274 25807 raft_consensus.cc:479] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.810204 25807 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.810565 25807 leader_election.cc:243] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.813426 25847 raft_consensus.cc:457] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.813731 25847 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.165s (starting election)
I0619 00:11:51.814169 25354 raft_consensus.cc:2784] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.814129 25847 raft_consensus.cc:479] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.815614 25847 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.815595 25082 catalog_manager.cc:3663] Got report from deleted tablet acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc): Sending delete request for this tablet
I0619 00:11:51.815801 25336 ts_tablet_manager.cc:1173] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.816049 25847 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.816578 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acc89522b693498c987ca5d8ef8275ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:51.817394 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:51.817975 25489 raft_consensus.cc:2206] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:51.818171 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:51.819988 25419 leader_election.cc:306] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.819749 25796 consensus_queue.cc:963] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I0619 00:11:51.820575 25725 tablet_bootstrap.cc:578] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:51.821878 25354 raft_consensus.cc:1081] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:51.822136 25727 raft_consensus.cc:628] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:51.822715 25727 consensus_queue.cc:227] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.823279 25796 consensus_queue.cc:963] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:51.826097 25736 ts_tablet_manager.cc:1369] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.827044 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:51.827775 25080 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 24 ms (attempt = 1)
W0619 00:11:51.828161 25080 catalog_manager.cc:3945] Failed to send 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.829154 25082 catalog_manager.cc:3663] Got report from deleted tablet 292b164659e343158de8e7e3eeb9be31 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0): Sending delete request for this tablet
I0619 00:11:51.831459 25470 tablet_service.cc:800] Processing DeleteTablet for tablet aaf62da07a9740c1aa44359d3f604513 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.829879 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:51.836261 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.838716 25727 ts_tablet_manager.cc:1073] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.000s
I0619 00:11:51.839092 25464 ts_tablet_manager.cc:1173] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.839221 25727 ts_tablet_manager.cc:1028] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:51.839323 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 40 ms (attempt = 2)
W0619 00:11:51.839593 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.840272 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 54373917567a479d93dc8a3fd006d2fe
I0619 00:11:51.840610 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.834925 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 94 ms (attempt = 3)
I0619 00:11:51.841042 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:51.841135 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:51.840324 25082 catalog_manager.cc:3663] Got report from deleted tablet 292b164659e343158de8e7e3eeb9be31 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0): Sending delete request for this tablet
I0619 00:11:51.841425 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.841928 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 292b164659e343158de8e7e3eeb9be31 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0)
W0619 00:11:51.842074 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.842900 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.842993 25290 leader_election.cc:393] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:51.844507 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:51.844655 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.822012 25323 tablet_service.cc:759] Processing CreateTablet for tablet e0070a115e8540cda5f999394deac616 (table=default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.791771 25724 ts_tablet_manager.cc:1073] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I0619 00:11:51.845700 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e0070a115e8540cda5f999394deac616. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.848372 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:51.849730 25727 tablet_bootstrap.cc:427] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:51.852337 25727 tablet_bootstrap.cc:578] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:51.852267 25326 ts_tablet_manager.cc:1173] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.852699 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:51.853334 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 76 ms (attempt = 2)
W0619 00:11:51.853603 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
W0619 00:11:51.853541 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times.
I0619 00:11:51.853750 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 292b164659e343158de8e7e3eeb9be31 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4c4769f75141f9b2ef864ab8a8d9e0) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.854017 25069 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 66 ms (attempt = 3)
W0619 00:11:51.854261 25069 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.854189 25601 ts_tablet_manager.cc:1173] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.855038 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:51.855093 25600 tablet_service.cc:759] Processing CreateTablet for tablet e0070a115e8540cda5f999394deac616 (table=default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.855438 25325 ts_tablet_manager.cc:1173] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.855710 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_e2e6deb5e5764586802390a904ba068f" }
I0619 00:11:51.856536 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.856618 25730 ts_tablet_manager.cc:1382] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.856886 25730 log.cc:986] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a85577b0beba4276b963e5c360544bc7
I0619 00:11:51.857357 25730 ts_tablet_manager.cc:1403] T a85577b0beba4276b963e5c360544bc7 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:51.858458 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "36f5eec972454080b7b434b4df7e62e3" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:51.859035 25624 raft_consensus.cc:2175] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:51.860203 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:51.860674 25622 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.345s (vote granted)
I0619 00:11:51.860913 25622 raft_consensus.cc:2245] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:51.862089 25423 leader_election.cc:379] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.862344 25423 leader_election.cc:270] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:51.862828 25796 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.044s (election complete)
I0619 00:11:51.863075 25796 raft_consensus.cc:2539] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:51.863226 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e0070a115e8540cda5f999394deac616. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.863291 25796 raft_consensus.cc:457] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:51.863569 25796 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.234s (starting election)
I0619 00:11:51.863803 25796 raft_consensus.cc:2784] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.865108 25724 ts_tablet_manager.cc:1028] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:51.865492 25724 tablet_bootstrap.cc:427] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:51.867300 25736 ts_tablet_manager.cc:1382] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:51.867584 25736 log.cc:986] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/aaf62da07a9740c1aa44359d3f604513
W0619 00:11:51.868755 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.869282 25736 ts_tablet_manager.cc:1403] T aaf62da07a9740c1aa44359d3f604513 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:51.870216 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:51.872781 25600 ts_tablet_manager.cc:1173] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.873149 25081 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 36f5eec972454080b7b434b4df7e62e3 (table default.table_e4aaded22fbb4c60a591e5ff45bfe2bc [id=122dad4ddc134d21b827dca8587ed257]). Expected version 1 got 0
I0619 00:11:51.873488 25290 leader_election.cc:393] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:51.874349 25290 leader_election.cc:270] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.870913 25796 raft_consensus.cc:479] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.875038 25796 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.871666 25724 tablet_bootstrap.cc:578] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.875967 25796 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.876494 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:51.876776 25807 raft_consensus.cc:2732] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.701s (election complete)
I0619 00:11:51.877014 25807 raft_consensus.cc:2487] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:51.878574 25419 leader_election.cc:306] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.878877 25081 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 36f5eec972454080b7b434b4df7e62e3 (table default.table_e4aaded22fbb4c60a591e5ff45bfe2bc [id=122dad4ddc134d21b827dca8587ed257]). Expected version 1 got 0
I0619 00:11:51.879427 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:51.879881 25622 raft_consensus.cc:2784] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.880509 25622 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.974s (vote granted)
I0619 00:11:51.880669 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 109 ms (attempt = 3)
I0619 00:11:51.854540 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 3b80a302aacb418091aee54ab4be7249 (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) successfully deleted
W0619 00:11:51.880975 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.882242 25463 tablet_service.cc:800] Processing DeleteTablet for tablet acc89522b693498c987ca5d8ef8275ea with delete_type TABLET_DATA_DELETED (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.882242 25470 tablet_service.cc:759] Processing CreateTablet for tablet e0070a115e8540cda5f999394deac616 (table=default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:51.882702 25467 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.882910 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e0070a115e8540cda5f999394deac616. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:51.884678 25807 raft_consensus.cc:457] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.884908 25807 raft_consensus.cc:2732] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.621s (starting election)
W0619 00:11:51.884937 25071 catalog_manager.cc:3255] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): alter failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]): Aborted: Transaction aborted
I0619 00:11:51.885247 25071 catalog_manager.cc:2891] Scheduling retry of 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 546 ms (attempt = 6)
I0619 00:11:51.885156 25807 raft_consensus.cc:479] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.885846 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.885648 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 53688f0e4a604bc4a07dec9278a8dad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:51.886106 25807 leader_election.cc:243] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.886266 25837 tablet.cc:1200] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:51.886471 25807 leader_election.cc:243] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:51.888504 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.887362 25622 raft_consensus.cc:2245] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:51.892138 25844 tablet.cc:1200] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
W0619 00:11:51.894412 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.895082 25730 tablet_replica.cc:261] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:51.895360 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.895603 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.895867 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.896106 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(72e28bc6b57442b782d8aa07e1152cf4)
W0619 00:11:51.896603 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.896365 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.896757 25423 leader_election.cc:379] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.897001 25423 leader_election.cc:270] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:51.897054 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.897305 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(72e28bc6b57442b782d8aa07e1152cf4)
I0619 00:11:51.897516 25730 raft_consensus.cc:2042] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.897835 25730 raft_consensus.cc:2069] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.898463 25796 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.832s (election complete)
I0619 00:11:51.899859 25796 raft_consensus.cc:2539] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:51.900789 25819 tablet.cc:1200] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:51.900565 25796 raft_consensus.cc:593] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:51.901971 25796 consensus_queue.cc:204] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.906103 25470 ts_tablet_manager.cc:1173] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:51.907678 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.909682 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.910166 25325 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.911895 25725 tablet_bootstrap.cc:427] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.903563 25080 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 36f5eec972454080b7b434b4df7e62e3 (table default.table_e4aaded22fbb4c60a591e5ff45bfe2bc [id=122dad4ddc134d21b827dca8587ed257]). Expected version 1 got 0
I0619 00:11:51.912221 25725 ts_tablet_manager.cc:1045] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.205s	user 0.004s	sys 0.005s
I0619 00:11:51.906008 25067 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 534 ms (attempt = 6)
W0619 00:11:51.912719 25067 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:51.913503 25725 raft_consensus.cc:325] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.914093 25725 raft_consensus.cc:351] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.914265 25725 raft_consensus.cc:628] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:51.914644 25081 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_e4aaded22fbb4c60a591e5ff45bfe2bc [id=122dad4ddc134d21b827dca8587ed257] alter complete (version 1)
I0619 00:11:51.915287 25725 consensus_queue.cc:227] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.917424 25725 ts_tablet_manager.cc:1073] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I0619 00:11:51.917989 25725 ts_tablet_manager.cc:1028] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:51.918342 25725 tablet_bootstrap.cc:427] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:51.920421 25725 tablet_bootstrap.cc:578] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:51.922945 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 101 ms (attempt = 3)
W0619 00:11:51.923233 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:51.923920 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_14bc9661c33c47beab02a4d149b07c42" } new_table_name: "default.table_824bd974d1ae4fc398285c3a2393333f" alter_external_catalogs: true
W0619 00:11:51.924937 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.924896 25069 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 153 ms (attempt = 4)
W0619 00:11:51.925158 25069 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
W0619 00:11:51.926220 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 52736.3us
I0619 00:11:51.927599 25741 ts_tablet_manager.cc:1382] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:51.927930 25741 log.cc:986] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/53688f0e4a604bc4a07dec9278a8dad6
I0619 00:11:51.928457 25741 ts_tablet_manager.cc:1403] T 53688f0e4a604bc4a07dec9278a8dad6 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:51.929337 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.930127 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 71 ms (attempt = 3)
I0619 00:11:51.930207 25730 ts_tablet_manager.cc:1369] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:51.930393 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.921192 25736 tablet_replica.cc:261] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:51.933105 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.933362 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.933645 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.933871 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.934128 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.934367 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.934641 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(53688f0e4a604bc4a07dec9278a8dad6)
I0619 00:11:51.934862 25736 raft_consensus.cc:2042] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.935154 25736 raft_consensus.cc:2069] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.936344 25736 ts_tablet_manager.cc:1369] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:51.937691 25087 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 51 ms (attempt = 1)
W0619 00:11:51.938028 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 has no consensus state
W0619 00:11:51.939152 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.941375 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 53688f0e4a604bc4a07dec9278a8dad6 (table default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]) successfully deleted
W0619 00:11:51.941830 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:51.941900 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
W0619 00:11:51.942410 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3b80a302aacb418091aee54ab4be7249 already in progress: creating tablet
I0619 00:11:51.942692 25067 catalog_manager.cc:2891] Scheduling retry of 3b80a302aacb418091aee54ab4be7249 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 554 ms (attempt = 6)
W0619 00:11:51.943115 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:51.943564 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) successfully deleted
I0619 00:11:51.944404 25323 ts_tablet_manager.cc:1173] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:51.944411 25071 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 2096 ms (attempt = 8)
I0619 00:11:51.943661 25847 raft_consensus.cc:457] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.950453 25847 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.645s (starting election)
I0619 00:11:51.950736 25847 raft_consensus.cc:479] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.951772 25847 leader_election.cc:243] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:51.952158 25847 leader_election.cc:243] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.950212 25725 tablet_bootstrap.cc:427] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:51.956207 25725 ts_tablet_manager.cc:1045] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.038s	user 0.004s	sys 0.002s
I0619 00:11:51.956643 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acc89522b693498c987ca5d8ef8275ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:51.957177 25623 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.947s (vote granted)
I0619 00:11:51.957409 25623 raft_consensus.cc:2245] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:51.953853 25087 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:51.958703 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 35acf55f35064c1491122a13ff9f5744
I0619 00:11:51.958988 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.959372 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.959758 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:51.960865 25290 leader_election.cc:306] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 292b164659e343158de8e7e3eeb9be31
W0619 00:11:51.961369 25290 leader_election.cc:306] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 292b164659e343158de8e7e3eeb9be31
I0619 00:11:51.961685 25290 leader_election.cc:270] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:51.961767 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:51.962455 25807 raft_consensus.cc:2732] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.567s (election complete)
I0619 00:11:51.962703 25807 raft_consensus.cc:2487] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:51.965575 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acc89522b693498c987ca5d8ef8275ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:51.967773 25724 tablet_bootstrap.cc:427] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:51.968147 25724 ts_tablet_manager.cc:1045] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.103s	user 0.005s	sys 0.004s
I0619 00:11:51.969626 25724 raft_consensus.cc:325] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.970320 25724 raft_consensus.cc:351] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.970533 25724 raft_consensus.cc:628] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:51.971427 25727 tablet_bootstrap.cc:427] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:51.971119 25724 consensus_queue.cc:227] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.971791 25727 ts_tablet_manager.cc:1045] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.122s	user 0.003s	sys 0.004s
I0619 00:11:51.972753 25087 catalog_manager.cc:3663] Got report from deleted tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:51.973467 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
W0619 00:11:51.973747 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.975442 25724 ts_tablet_manager.cc:1073] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
W0619 00:11:51.975917 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:51.973881 25727 raft_consensus.cc:325] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.976843 25727 raft_consensus.cc:351] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.977015 25727 raft_consensus.cc:628] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
W0619 00:11:51.944871 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:51.977468 25727 consensus_queue.cc:227] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.976263 25724 ts_tablet_manager.cc:1028] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:51.980804 25080 catalog_manager.cc:3663] Got report from deleted tablet 3b80a302aacb418091aee54ab4be7249 (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet acc89522b693498c987ca5d8ef8275ea): Sending delete request for this tablet
I0619 00:11:51.981209 25727 ts_tablet_manager.cc:1073] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I0619 00:11:51.981521 25727 ts_tablet_manager.cc:1028] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:51.981497 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:51.983777 25727 tablet_bootstrap.cc:427] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:51.984947 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 174 ms (attempt = 4)
W0619 00:11:51.985221 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:51.985713 25724 tablet_bootstrap.cc:427] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:51.986718 25081 catalog_manager.cc:3663] Got report from deleted tablet 3b80a302aacb418091aee54ab4be7249 (table default.table_00f27d6b83d24c4e8088e0b0ac0d0cca [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet acc89522b693498c987ca5d8ef8275ea): Sending delete request for this tablet
I0619 00:11:51.987365 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:51.989111 25724 tablet_bootstrap.cc:578] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:51.989990 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 68 ms (attempt = 2)
W0619 00:11:51.990259 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 has no consensus state
I0619 00:11:51.990325 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:51.990674 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 158 ms (attempt = 4)
I0619 00:11:51.957530 25725 raft_consensus.cc:325] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:51.990916 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:51.991012 25725 raft_consensus.cc:351] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:51.991325 25748 fs_manager.cc:417] Time spent opening block manager: real 0.032s	user 0.001s	sys 0.002s
I0619 00:11:51.991278 25725 raft_consensus.cc:628] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:51.958446 25423 leader_election.cc:379] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.991643 25423 leader_election.cc:270] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:11:51.991924 25419 leader_election.cc:306] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:51.992312 25847 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.643s (election complete)
I0619 00:11:51.992563 25741 tablet_replica.cc:261] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:51.992559 25847 raft_consensus.cc:2539] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:51.992823 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.992784 25847 raft_consensus.cc:457] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:51.993014 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.993001 25847 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.554s (starting election)
I0619 00:11:51.993194 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.993194 25847 raft_consensus.cc:2784] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:51.991528 25748 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
uuid: "b2a016cc9f274bc1979068306ec6d15e"
format_stamp: "Formatted at 2018-06-19 07:11:44 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:11:51.993379 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.993590 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.993811 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.993530 25748 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:11:51.993985 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(292b164659e343158de8e7e3eeb9be31)
I0619 00:11:51.994150 25741 raft_consensus.cc:2042] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:51.994417 25741 raft_consensus.cc:2069] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:51.996225 25807 raft_consensus.cc:457] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:51.996436 25807 raft_consensus.cc:2732] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.088s (starting election)
I0619 00:11:51.996688 25807 raft_consensus.cc:479] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:51.997606 25807 leader_election.cc:243] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:51.998344 25807 leader_election.cc:243] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:51.999227 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b6fb03b2b1ed49ab94d4e017c1ba1408"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.000200 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b6fb03b2b1ed49ab94d4e017c1ba1408 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.001864 25082 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.001696 25847 raft_consensus.cc:479] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.002653 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 168 ms (attempt = 4)
I0619 00:11:52.002718 25847 leader_election.cc:243] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:52.002916 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.003188 25847 leader_election.cc:243] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.004879 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acc89522b693498c987ca5d8ef8275ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:52.003402 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:52.006145 25419 leader_election.cc:306] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:52.006402 25071 catalog_manager.cc:3249] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): alter failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I0619 00:11:52.009907 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.010185 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_00f27d6b83d24c4e8088e0b0ac0d0cca" } new_table_name: "default.table_48d04a9a6f224dd983111f0215fb417b" alter_external_catalogs: true
I0619 00:11:52.010627 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.011134 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet e3708e05f6054306b5bf7f1df2140e3d from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:51.994355 25730 ts_tablet_manager.cc:1382] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I0619 00:11:52.011507 25730 log.cc:986] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.004845 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acc89522b693498c987ca5d8ef8275ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:52.007391 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.008258 25601 tablet_service.cc:759] Processing CreateTablet for tablet d6c3c7f783cb43d59b46502261aeced3 (table=default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.011550 25470 tablet_service.cc:759] Processing CreateTablet for tablet d6c3c7f783cb43d59b46502261aeced3 (table=default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.011811 25623 raft_consensus.cc:2784] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.009356 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.012126 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d6c3c7f783cb43d59b46502261aeced3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.012126 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d6c3c7f783cb43d59b46502261aeced3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.012074 25623 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.957s (vote granted)
I0619 00:11:51.999537 25811 raft_consensus.cc:457] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.012684 25811 raft_consensus.cc:2732] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.956s (starting election)
I0619 00:11:52.012933 25811 raft_consensus.cc:479] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.013880 25811 leader_election.cc:243] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.014259 25811 leader_election.cc:243] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.009021 25598 tablet_service.cc:759] Processing CreateTablet for tablet edfdedbd4eca4b62be24b1649f820d0e (table=default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.016263 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet edfdedbd4eca4b62be24b1649f820d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.011660 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) successfully deleted
I0619 00:11:52.016918 25623 raft_consensus.cc:2245] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:52.017302 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c18355df78d344839a56189c7549ab1f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.017907 25730 ts_tablet_manager.cc:1403] T 72e28bc6b57442b782d8aa07e1152cf4 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:11:52.018535 25553 leader_election.cc:306] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.018498 25464 tablet_service.cc:759] Processing CreateTablet for tablet edfdedbd4eca4b62be24b1649f820d0e (table=default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.019146 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet edfdedbd4eca4b62be24b1649f820d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.019755 25325 tablet_service.cc:759] Processing CreateTablet for tablet d6c3c7f783cb43d59b46502261aeced3 (table=default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.020313 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d6c3c7f783cb43d59b46502261aeced3. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.021241 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.022313 25323 tablet_service.cc:759] Processing CreateTablet for tablet edfdedbd4eca4b62be24b1649f820d0e (table=default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.022985 25082 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.023633 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:52.026872 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet cb57d196f0724247b6a7f9a34be8377a already in progress: creating tablet
I0619 00:11:52.027118 25067 catalog_manager.cc:2891] Scheduling retry of cb57d196f0724247b6a7f9a34be8377a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 561 ms (attempt = 6)
I0619 00:11:52.028311 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c18355df78d344839a56189c7549ab1f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.029709 25468 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.029645 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 102) took 1004ms. Request Metrics: {"tablet-delete.queue_time_us":868820}
I0619 00:11:52.030354 25423 leader_election.cc:379] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.030606 25730 tablet_replica.cc:261] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:52.030666 25423 leader_election.cc:270] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:52.030843 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.031069 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.031286 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.031507 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.031787 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.031853 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet edfdedbd4eca4b62be24b1649f820d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.032040 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.032258 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.032454 25730 raft_consensus.cc:2042] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.032817 25730 raft_consensus.cc:2069] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.034032 25730 ts_tablet_manager.cc:1369] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:52.034025 25558 leader_election.cc:306] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.034332 25558 leader_election.cc:270] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:52.034736 25558 raft_consensus.cc:2442] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:11:52.035249 25725 consensus_queue.cc:227] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.036231 25796 raft_consensus.cc:2732] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.540s (election complete)
I0619 00:11:52.036502 25796 raft_consensus.cc:2539] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:52.036813 25796 raft_consensus.cc:593] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:52.037034 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet e3708e05f6054306b5bf7f1df2140e3d
I0619 00:11:52.037334 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e3708e05f6054306b5bf7f1df2140e3d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:11:52.037221 25796 consensus_queue.cc:204] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.037739 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e3708e05f6054306b5bf7f1df2140e3d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:11:52.038462 25725 ts_tablet_manager.cc:1073] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.082s	user 0.004s	sys 0.000s
I0619 00:11:52.038879 25725 ts_tablet_manager.cc:1028] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:52.038097 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e3708e05f6054306b5bf7f1df2140e3d on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:11:52.039520 25081 catalog_manager.cc:4596] ReplaceTablet: tablet e3708e05f6054306b5bf7f1df2140e3d deleted and replaced by tablet 9d44681acacf4f04b6fcfbd4c9a18e87
I0619 00:11:52.039217 25725 tablet_bootstrap.cc:427] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:52.042168 25725 tablet_bootstrap.cc:578] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:52.042919 25601 ts_tablet_manager.cc:1173] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.043812 25081 catalog_manager.cc:3663] Got report from deleted tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.044450 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.045164 25069 catalog_manager.cc:2891] Scheduling retry of 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC with a delay of 513 ms (attempt = 6)
W0619 00:11:52.045435 25069 catalog_manager.cc:2910] Async tablet task 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a88e394b199431e8b794a1b1aec847f consensus state has no leader
I0619 00:11:52.047237 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.047813 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.048606 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.049183 25290 leader_election.cc:306] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 3b80a302aacb418091aee54ab4be7249
I0619 00:11:52.049824 25336 tablet_service.cc:800] Processing DeleteTablet for tablet e3708e05f6054306b5bf7f1df2140e3d with delete_type TABLET_DATA_DELETED (replaced by tablet 9d44681acacf4f04b6fcfbd4c9a18e87) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.050365 25290 leader_election.cc:306] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 3b80a302aacb418091aee54ab4be7249
I0619 00:11:52.050359 25087 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 28 ms (attempt = 1)
W0619 00:11:52.050659 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 981d862d1bab4b14bb10e6fc2174e1cc consensus state has no leader
I0619 00:11:52.050693 25290 leader_election.cc:270] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.051225 25807 raft_consensus.cc:2732] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.246s (election complete)
I0619 00:11:52.051445 25807 raft_consensus.cc:2487] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.055091 25598 ts_tablet_manager.cc:1173] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:52.058444 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3b80a302aacb418091aee54ab4be7249 because the tablet was not found. No further retry: Not found: Tablet not found: 3b80a302aacb418091aee54ab4be7249
I0619 00:11:52.058864 25727 tablet_bootstrap.cc:578] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.058639 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_6cd711ffd03644e8a98c73f5d95fa03f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:52.059427 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 103 ms (attempt = 3)
W0619 00:11:52.059564 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6cd711ffd03644e8a98c73f5d95fa03f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:52.059722 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 has no consensus state
I0619 00:11:52.060348 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 132 ms (attempt = 4)
W0619 00:11:52.060605 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.060885 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) successfully deleted
I0619 00:11:52.062834 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_debbb4b8c339468c97105946dd3b40f4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.063685 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_debbb4b8c339468c97105946dd3b40f4 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.065959 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 560 ms (attempt = 6)
W0619 00:11:52.044837 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.066220 25067 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_24cb60c44bca436082a96bb044ee559b [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:52.066241 25741 ts_tablet_manager.cc:1369] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.069196 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet 292b164659e343158de8e7e3eeb9be31 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:52.069375 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.052060 25736 ts_tablet_manager.cc:1382] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.069867 25736 log.cc:986] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/53688f0e4a604bc4a07dec9278a8dad6
I0619 00:11:52.070361 25736 ts_tablet_manager.cc:1403] T 53688f0e4a604bc4a07dec9278a8dad6 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:52.062713 25602 tablet_service.cc:800] Processing DeleteTablet for tablet e3708e05f6054306b5bf7f1df2140e3d with delete_type TABLET_DATA_DELETED (replaced by tablet 9d44681acacf4f04b6fcfbd4c9a18e87) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.062763 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.073407 25724 tablet_bootstrap.cc:427] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.078212 25323 ts_tablet_manager.cc:1173] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.078529 25724 ts_tablet_manager.cc:1045] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.093s	user 0.004s	sys 0.000s
I0619 00:11:52.079458 25069 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 278 ms (attempt = 5)
W0619 00:11:52.079736 25069 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.080283 25724 raft_consensus.cc:325] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.081100 25724 raft_consensus.cc:351] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.081154 25325 ts_tablet_manager.cc:1173] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.081303 25724 raft_consensus.cc:628] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.081357 25067 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 60 ms (attempt = 2)
W0619 00:11:52.081622 25067 catalog_manager.cc:2910] Async tablet task 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 981d862d1bab4b14bb10e6fc2174e1cc consensus state has no leader
I0619 00:11:52.081820 25724 consensus_queue.cc:227] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.055454 25464 ts_tablet_manager.cc:1173] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.083360 25724 ts_tablet_manager.cc:1073] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I0619 00:11:52.083741 25724 ts_tablet_manager.cc:1028] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.084111 25724 tablet_bootstrap.cc:427] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:52.085535 25811 raft_consensus.cc:457] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.085798 25811 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.573s (starting election)
I0619 00:11:52.086040 25811 raft_consensus.cc:479] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.086503 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.086501 25736 tablet_replica.cc:261] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:52.086761 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.086963 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.087023 25811 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.087150 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.087334 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.087424 25811 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.087550 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.088167 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.088373 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.088332 25068 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 1056 ms (attempt = 7)
W0619 00:11:52.088604 25068 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.088552 25736 raft_consensus.cc:2042] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:52.088912 25736 consensus_peers.cc:456] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:52.089148 25736 consensus_peers.cc:456] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:52.089828 25736 pending_rounds.cc:63] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Trying to abort 1 pending transactions.
I0619 00:11:52.090018 25736 pending_rounds.cc:68] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390910110580736 op_type: NO_OP noop_request { }
I0619 00:11:52.090358 25736 raft_consensus.cc:2622] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:52.090291 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.090694 25736 raft_consensus.cc:2069] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:52.091842 25553 leader_election.cc:306] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:52.092319 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.093624 25080 catalog_manager.cc:3663] Got report from deleted tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.093813 25724 tablet_bootstrap.cc:578] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:52.094020 25080 catalog_manager.cc:3663] Got report from deleted tablet acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc): Sending delete request for this tablet
I0619 00:11:52.094514 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) successfully deleted
I0619 00:11:52.094776 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c3660a004537433aaff9ead44b4615e4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.095701 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c3660a004537433aaff9ead44b4615e4 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.095824 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_1563f330df024855ab3d01187bdc7661" }
I0619 00:11:52.096181 25727 tablet_bootstrap.cc:427] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.096526 25727 ts_tablet_manager.cc:1045] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.113s	user 0.005s	sys 0.000s
W0619 00:11:52.097867 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.097812 25727 raft_consensus.cc:325] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.098333 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:52.098402 25727 raft_consensus.cc:351] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.098625 25727 raft_consensus.cc:628] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.099185 25727 consensus_queue.cc:227] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.102308 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.104677 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.105744 25727 ts_tablet_manager.cc:1073] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I0619 00:11:52.107244 25727 ts_tablet_manager.cc:1028] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:52.107617 25727 tablet_bootstrap.cc:427] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:52.111636 25325 tablet_service.cc:759] Processing CreateTablet for tablet 0a7470bfeab041f9895e491231b57eb7 (table=default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.112243 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a7470bfeab041f9895e491231b57eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.112534 25730 ts_tablet_manager.cc:1382] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.112874 25730 log.cc:986] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/c18355df78d344839a56189c7549ab1f
I0619 00:11:52.113382 25730 ts_tablet_manager.cc:1403] T c18355df78d344839a56189c7549ab1f P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:52.114886 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 105) took 1034ms. Request Metrics: {"tablet-delete.queue_time_us":949743}
I0619 00:11:52.115852 25727 tablet_bootstrap.cc:578] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.118557 25730 tablet_replica.cc:261] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:52.119304 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.120295 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.120731 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.121192 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.122119 25727 tablet_bootstrap.cc:427] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.092140 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.095688 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.122786 25811 raft_consensus.cc:457] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.122954 25727 ts_tablet_manager.cc:1045] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I0619 00:11:52.123030 25811 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.640s (starting election)
I0619 00:11:52.123275 25811 raft_consensus.cc:479] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.098675 25464 tablet_service.cc:800] Processing DeleteTablet for tablet e3708e05f6054306b5bf7f1df2140e3d with delete_type TABLET_DATA_DELETED (replaced by tablet 9d44681acacf4f04b6fcfbd4c9a18e87) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.102838 25470 ts_tablet_manager.cc:1173] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.102802 25083 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:52.124299 25811 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.105710 25796 raft_consensus.cc:457] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.111871 25725 tablet_bootstrap.cc:427] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.124704 25811 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.124722 25796 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.831s (starting election)
I0619 00:11:52.124812 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.124999 25725 ts_tablet_manager.cc:1045] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.086s	user 0.004s	sys 0.001s
I0619 00:11:52.124999 25796 raft_consensus.cc:479] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.126057 25796 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.126467 25796 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.126905 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.126881 25558 leader_election.cc:306] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.127147 25558 leader_election.cc:270] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.127228 25071 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 71 ms (attempt = 3)
I0619 00:11:52.128010 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) successfully deleted
I0619 00:11:52.106319 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.128727 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.128931 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acc89522b693498c987ca5d8ef8275ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:52.130117 25725 raft_consensus.cc:325] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.130748 25725 raft_consensus.cc:351] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:52.130906 25553 leader_election.cc:306] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.130939 25725 raft_consensus.cc:628] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:52.131778 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.132212 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb57d196f0724247b6a7f9a34be8377a" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.131464 25725 consensus_queue.cc:227] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.133543 25811 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.523s (election complete)
I0619 00:11:52.133797 25811 raft_consensus.cc:2487] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:52.133841 25419 leader_election.cc:306] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.133275 25080 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:52.134414 25725 ts_tablet_manager.cc:1073] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.001s
I0619 00:11:52.134740 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:52.128473 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
I0619 00:11:52.135347 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 70 ms (attempt = 2)
I0619 00:11:52.136202 25725 ts_tablet_manager.cc:1028] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
W0619 00:11:52.136584 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet aaf62da07a9740c1aa44359d3f604513 because the tablet was not found. No further retry: Not found: Tablet not found: aaf62da07a9740c1aa44359d3f604513
I0619 00:11:52.136720 25725 tablet_bootstrap.cc:427] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:52.137303 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.137254 25601 tablet_service.cc:759] Processing CreateTablet for tablet 0a7470bfeab041f9895e491231b57eb7 (table=default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:52.137595 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a4c653a255984d3bac88df63362b9334 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a4c653a255984d3bac88df63362b9334 already in progress: creating tablet
I0619 00:11:52.137837 25071 catalog_manager.cc:2891] Scheduling retry of a4c653a255984d3bac88df63362b9334 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 42 ms (attempt = 1)
I0619 00:11:52.137902 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a7470bfeab041f9895e491231b57eb7. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.138202 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a52cbcbe35374d518bca1e0fc24d88d4 already in progress: creating tablet
I0619 00:11:52.138406 25071 catalog_manager.cc:2891] Scheduling retry of a52cbcbe35374d518bca1e0fc24d88d4 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 69 ms (attempt = 2)
W0619 00:11:52.138795 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
W0619 00:11:52.139204 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:52.139407 25071 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 79 ms (attempt = 2)
I0619 00:11:52.140291 25470 tablet_service.cc:759] Processing CreateTablet for tablet 0a7470bfeab041f9895e491231b57eb7 (table=default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.140748 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet edfdedbd4eca4b62be24b1649f820d0e
I0619 00:11:52.140892 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a7470bfeab041f9895e491231b57eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.141029 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.141398 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.141790 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.140367 25080 catalog_manager.cc:3663] Got report from deleted tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b): Sending delete request for this tablet
I0619 00:11:52.142299 25323 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.142894 25067 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 84 ms (attempt = 3)
W0619 00:11:52.143172 25067 catalog_manager.cc:2910] Async tablet task 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 981d862d1bab4b14bb10e6fc2174e1cc consensus state has no leader
I0619 00:11:52.145706 25741 ts_tablet_manager.cc:1382] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.145967 25741 log.cc:986] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/292b164659e343158de8e7e3eeb9be31
I0619 00:11:52.146414 25741 ts_tablet_manager.cc:1403] T 292b164659e343158de8e7e3eeb9be31 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:52.146574 25725 tablet_bootstrap.cc:578] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:52.147626 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.148908 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:52.148991 25423 leader_election.cc:306] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
W0619 00:11:52.149685 25558 leader_election.cc:306] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.149821 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 295 ms (attempt = 5)
I0619 00:11:52.149971 25558 leader_election.cc:270] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:52.150120 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.150437 25811 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.533s (election complete)
I0619 00:11:52.150656 25811 raft_consensus.cc:2487] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.150749 25423 leader_election.cc:270] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.150913 25601 ts_tablet_manager.cc:1173] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.151196 25796 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.816s (election complete)
I0619 00:11:52.151414 25796 raft_consensus.cc:2487] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.151881 25811 raft_consensus.cc:457] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.152094 25811 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.905s (starting election)
I0619 00:11:52.152320 25811 raft_consensus.cc:479] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.153244 25464 tablet_service.cc:800] Processing DeleteTablet for tablet acc89522b693498c987ca5d8ef8275ea with delete_type TABLET_DATA_DELETED (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.153270 25811 leader_election.cc:243] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.153633 25811 leader_election.cc:243] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:52.153744 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.154640 25736 ts_tablet_manager.cc:1369] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.142937 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:52.155191 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "397ae10bb6d841b4ba561ec431d29864" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.155406 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "397ae10bb6d841b4ba561ec431d29864" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:52.156504 25558 leader_election.cc:306] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.157369 25807 raft_consensus.cc:457] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.157630 25807 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.568s (starting election)
W0619 00:11:52.157845 25553 leader_election.cc:306] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.158092 25553 leader_election.cc:270] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.157871 25807 raft_consensus.cc:479] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.158843 25807 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.158879 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 1068 ms (attempt = 7)
W0619 00:11:52.159128 25067 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.159224 25807 leader_election.cc:243] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.160562 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.161118 25489 raft_consensus.cc:2206] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:52.160995 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "35acf55f35064c1491122a13ff9f5744" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:52.161568 25623 raft_consensus.cc:2175] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:52.162281 25290 leader_election.cc:393] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:52.163033 25290 leader_election.cc:393] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:52.163271 25290 leader_election.cc:270] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.163902 25811 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.715s (election complete)
I0619 00:11:52.163936 25807 raft_consensus.cc:2732] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.690s (election complete)
I0619 00:11:52.164144 25807 raft_consensus.cc:2784] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.164144 25811 raft_consensus.cc:2487] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.164254 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 133 ms (attempt = 4)
I0619 00:11:52.164413 25811 raft_consensus.cc:457] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:52.164523 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:52.122627 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.164633 25811 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.943s (starting election)
I0619 00:11:52.164903 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.165181 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.165385 25730 raft_consensus.cc:2042] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.165067 25811 raft_consensus.cc:479] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.165488 25080 catalog_manager.cc:3663] Got report from deleted tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b): Sending delete request for this tablet
I0619 00:11:52.165766 25730 raft_consensus.cc:2069] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.166101 25811 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.166244 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:52.166533 25811 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.167090 25730 ts_tablet_manager.cc:1369] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:52.167974 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.168200 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.169656 25601 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:52.169672 25553 leader_election.cc:306] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.170117 25727 raft_consensus.cc:325] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.170557 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 53688f0e4a604bc4a07dec9278a8dad6 (table default.table_15724e9cddcb457a8edaebf78d6afc15 [id=bd7b4ec53e7b418a968d1a7e3dd368ed]) successfully deleted
I0619 00:11:52.170812 25727 raft_consensus.cc:351] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.171026 25727 raft_consensus.cc:628] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.172097 25724 tablet_bootstrap.cc:427] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.172430 25724 ts_tablet_manager.cc:1045] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.088s	user 0.004s	sys 0.003s
I0619 00:11:52.173954 25724 raft_consensus.cc:325] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.174548 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 270 ms (attempt = 5)
I0619 00:11:52.174685 25724 raft_consensus.cc:351] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:52.174840 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.174932 25724 raft_consensus.cc:628] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.175498 25724 consensus_queue.cc:227] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.177052 25807 raft_consensus.cc:2487] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.178155 25724 ts_tablet_manager.cc:1073] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
W0619 00:11:52.178702 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.179132 25724 ts_tablet_manager.cc:1028] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.179069 25464 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.179510 25724 tablet_bootstrap.cc:427] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:52.179877 25464 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.181159 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:52.181454 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.181591 25325 ts_tablet_manager.cc:1173] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.181725 25727 consensus_queue.cc:227] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.182687 25464 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:52.184355 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.185127 25558 leader_election.cc:306] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.185398 25558 leader_election.cc:270] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.186043 25811 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.925s (election complete)
I0619 00:11:52.186264 25811 raft_consensus.cc:2487] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:52.188514 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.188760 25323 tablet_service.cc:759] Processing CreateTablet for tablet 1c4c69294cb945938004e5b9a0550e72 (table=default.table_6cd711ffd03644e8a98c73f5d95fa03f [id=9c66376e1d444a928cdf40dbb2292f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.190229 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
I0619 00:11:52.191017 25080 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 48 ms (attempt = 1)
W0619 00:11:52.191355 25080 catalog_manager.cc:3945] Failed to send 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:52.191431 25336 tablet_service.cc:759] Processing CreateTablet for tablet fc534f75faf345f487abeda223ad88db (table=default.table_debbb4b8c339468c97105946dd3b40f4 [id=0aca9e47073f4ac29960cd82f4a54ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.193761 25727 ts_tablet_manager.cc:1073] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.071s	user 0.004s	sys 0.000s
I0619 00:11:52.195652 25724 tablet_bootstrap.cc:578] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:52.198271 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.198329 25470 ts_tablet_manager.cc:1173] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.198694 25796 raft_consensus.cc:457] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.198778 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 279 ms (attempt = 5)
I0619 00:11:52.199045 25796 raft_consensus.cc:2732] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.859s (starting election)
W0619 00:11:52.199167 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:52.199584 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 275 ms (attempt = 5)
I0619 00:11:52.199307 25796 raft_consensus.cc:479] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.199862 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.200336 25796 leader_election.cc:243] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.201086 25326 tablet_service.cc:759] Processing CreateTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table=default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.201236 25796 leader_election.cc:243] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.201709 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9d44681acacf4f04b6fcfbd4c9a18e87. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.202009 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f372b9fb174844eca1662c62039a3c71" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:52.203450 25419 leader_election.cc:306] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.203574 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 292b164659e343158de8e7e3eeb9be31 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) successfully deleted
I0619 00:11:52.204007 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc534f75faf345f487abeda223ad88db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.204267 25741 tablet_replica.cc:261] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:52.204556 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(aaf62da07a9740c1aa44359d3f604513)
W0619 00:11:52.204567 25423 leader_election.cc:306] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
I0619 00:11:52.204841 25423 leader_election.cc:270] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.204864 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.205112 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.205343 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.205437 25796 raft_consensus.cc:2732] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.790s (election complete)
I0619 00:11:52.205646 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.205677 25796 raft_consensus.cc:2487] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.205919 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.206140 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(aaf62da07a9740c1aa44359d3f604513)
I0619 00:11:52.206346 25741 raft_consensus.cc:2042] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.206658 25741 raft_consensus.cc:2069] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.207825 25741 ts_tablet_manager.cc:1369] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.204988 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1c4c69294cb945938004e5b9a0550e72. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.209511 25727 ts_tablet_manager.cc:1028] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:52.209779 25080 catalog_manager.cc:3663] Got report from deleted tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (replaced by tablet a85577b0beba4276b963e5c360544bc7): Sending delete request for this tablet
I0619 00:11:52.209903 25727 tablet_bootstrap.cc:427] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:52.210508 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:52.212313 25068 catalog_manager.cc:2891] Scheduling retry of acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 1060 ms (attempt = 7)
W0619 00:11:52.212594 25068 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:52.212673 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.215720 25325 tablet_service.cc:759] Processing CreateTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 (table=default.table_b6fb03b2b1ed49ab94d4e017c1ba1408 [id=9249a51147764492bf0fb06b5c500da7]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.216048 25326 ts_tablet_manager.cc:1173] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.217393 25727 tablet_bootstrap.cc:578] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.217329 25080 catalog_manager.cc:3663] Got report from deleted tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (replaced by tablet a85577b0beba4276b963e5c360544bc7): Sending delete request for this tablet
I0619 00:11:52.218063 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:52.200440 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.222862 25601 tablet_service.cc:759] Processing CreateTablet for tablet 1c4c69294cb945938004e5b9a0550e72 (table=default.table_6cd711ffd03644e8a98c73f5d95fa03f [id=9c66376e1d444a928cdf40dbb2292f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.223482 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1c4c69294cb945938004e5b9a0550e72. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.223541 25464 tablet_service.cc:759] Processing CreateTablet for tablet fc534f75faf345f487abeda223ad88db (table=default.table_debbb4b8c339468c97105946dd3b40f4 [id=0aca9e47073f4ac29960cd82f4a54ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.223930 25725 tablet_bootstrap.cc:427] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.224156 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc534f75faf345f487abeda223ad88db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.224244 25725 ts_tablet_manager.cc:1045] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.088s	user 0.004s	sys 0.001s
I0619 00:11:52.224757 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.225133 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.226354 25602 tablet_service.cc:759] Processing CreateTablet for tablet fc534f75faf345f487abeda223ad88db (table=default.table_debbb4b8c339468c97105946dd3b40f4 [id=0aca9e47073f4ac29960cd82f4a54ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.226943 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc534f75faf345f487abeda223ad88db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.227941 25468 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.228332 25796 raft_consensus.cc:457] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.229025 25796 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.529s (starting election)
I0619 00:11:52.226074 25725 raft_consensus.cc:325] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.229365 25067 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 143 ms (attempt = 4)
I0619 00:11:52.229570 25725 raft_consensus.cc:351] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.229260 25796 raft_consensus.cc:479] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.229655 25067 catalog_manager.cc:2910] Async tablet task 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 981d862d1bab4b14bb10e6fc2174e1cc consensus state has no leader
I0619 00:11:52.229780 25725 raft_consensus.cc:628] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:52.230321 25725 consensus_queue.cc:227] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.234395 25082 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:52.235054 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.236522 25602 ts_tablet_manager.cc:1173] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.236624 25464 ts_tablet_manager.cc:1173] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.237457 25080 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_24cb60c44bca436082a96bb044ee559b" } new_table_name: "default.table_cc45383880634c41948247ee9fa098d1" alter_external_catalogs: true
W0619 00:11:52.238842 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.238421 25470 tablet_service.cc:759] Processing CreateTablet for tablet 1c4c69294cb945938004e5b9a0550e72 (table=default.table_6cd711ffd03644e8a98c73f5d95fa03f [id=9c66376e1d444a928cdf40dbb2292f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.239359 25847 raft_consensus.cc:457] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.239631 25847 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.808s (starting election)
I0619 00:11:52.240011 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 49 ms (attempt = 2)
I0619 00:11:52.239881 25847 raft_consensus.cc:479] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.240276 25082 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
W0619 00:11:52.240537 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.240859 25847 leader_election.cc:243] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.240926 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.241279 25847 leader_election.cc:243] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.241364 25796 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:52.240276 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:52.241840 25796 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.240538 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1c4c69294cb945938004e5b9a0550e72. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.245743 25725 ts_tablet_manager.cc:1073] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I0619 00:11:52.246119 25725 ts_tablet_manager.cc:1028] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:52.246480 25725 tablet_bootstrap.cc:427] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:52.247802 25423 leader_election.cc:306] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:52.248692 25601 ts_tablet_manager.cc:1173] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.219177 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e5e691e6d374cd4bba1f18d8c821723. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.224196 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.229284 25336 ts_tablet_manager.cc:1173] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.252213 25323 ts_tablet_manager.cc:1173] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.252681 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a85577b0beba4276b963e5c360544bc7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.252725 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03522c48c50943b38cae5aa88562e583" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:52.254318 25419 leader_election.cc:306] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:52.254997 25419 leader_election.cc:306] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.255241 25419 leader_election.cc:270] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.255681 25796 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.614s (election complete)
I0619 00:11:52.255735 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_2c7f2c73082443ab869e2910097fa796" }
I0619 00:11:52.255903 25796 raft_consensus.cc:2487] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.261144 25725 tablet_bootstrap.cc:578] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:52.255133 25736 ts_tablet_manager.cc:1382] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:52.262171 25736 log.cc:986] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/acc89522b693498c987ca5d8ef8275ea
I0619 00:11:52.261303 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.268811 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:52.269121 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.270085 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.270478 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.271106 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.273514 25080 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 50 ms (attempt = 1)
W0619 00:11:52.273926 25080 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:52.273919 25468 tablet_service.cc:759] Processing CreateTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 (table=default.table_b6fb03b2b1ed49ab94d4e017c1ba1408 [id=9249a51147764492bf0fb06b5c500da7]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.251722 25602 tablet_service.cc:759] Processing CreateTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table=default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.274585 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e5e691e6d374cd4bba1f18d8c821723. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.274982 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9d44681acacf4f04b6fcfbd4c9a18e87. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.272758 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.272702 25598 tablet_service.cc:759] Processing CreateTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 (table=default.table_b6fb03b2b1ed49ab94d4e017c1ba1408 [id=9249a51147764492bf0fb06b5c500da7]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.275847 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e5e691e6d374cd4bba1f18d8c821723. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.264674 25727 tablet_bootstrap.cc:427] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.278259 25727 ts_tablet_manager.cc:1045] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.068s	user 0.003s	sys 0.002s
I0619 00:11:52.279893 25741 ts_tablet_manager.cc:1382] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.280164 25741 log.cc:986] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/aaf62da07a9740c1aa44359d3f604513
I0619 00:11:52.280488 25736 ts_tablet_manager.cc:1403] T acc89522b693498c987ca5d8ef8275ea P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:52.280717 25741 ts_tablet_manager.cc:1403] T aaf62da07a9740c1aa44359d3f604513 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:52.282486 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) successfully deleted
I0619 00:11:52.283411 25468 ts_tablet_manager.cc:1173] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:52.284698 25423 leader_election.cc:306] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:52.284955 25423 leader_election.cc:270] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.285385 25796 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.631s (election complete)
I0619 00:11:52.285704 25796 raft_consensus.cc:2487] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.285876 25467 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.286303 25463 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.286579 25462 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.279639 25727 raft_consensus.cc:325] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.286931 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.287073 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.286721 25467 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.287883 25736 tablet_replica.cc:261] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:52.287212 25727 raft_consensus.cc:351] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.289599 25727 raft_consensus.cc:628] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.290508 25727 consensus_queue.cc:227] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.291254 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 113 ms (attempt = 3)
W0619 00:11:52.291541 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:52.292040 25724 tablet_bootstrap.cc:427] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.292382 25724 ts_tablet_manager.cc:1045] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.113s	user 0.007s	sys 0.001s
I0619 00:11:52.293030 25598 ts_tablet_manager.cc:1173] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.294034 25724 raft_consensus.cc:325] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.294735 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.294970 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.295186 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.295401 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.295675 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.295899 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.296109 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.296304 25736 raft_consensus.cc:2042] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.296602 25736 raft_consensus.cc:2069] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.297382 25468 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.291550 25464 tablet_service.cc:759] Processing CreateTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table=default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.298300 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 280 ms (attempt = 5)
W0619 00:11:52.298553 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:52.289342 25725 tablet_bootstrap.cc:427] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.298867 25724 raft_consensus.cc:351] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.299052 25724 raft_consensus.cc:628] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.299182 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) successfully deleted
I0619 00:11:52.299660 25724 consensus_queue.cc:227] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.300803 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_84d38522d00a488c82a136b79665c4cd" } new_table_name: "default.table_caad097bfc684f938c5e58f0eaa18870" alter_external_catalogs: true
I0619 00:11:52.303642 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet 35acf55f35064c1491122a13ff9f5744 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:52.304718 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.307358 25083 catalog_manager.cc:3663] Got report from deleted tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
W0619 00:11:52.307670 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.308040 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.308725 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3828fbda3c26454f85595d8e6fbb70e5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.309588 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3828fbda3c26454f85595d8e6fbb70e5 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:52.309660 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.311729 25724 ts_tablet_manager.cc:1073] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.000s
I0619 00:11:52.312163 25724 ts_tablet_manager.cc:1028] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.312530 25724 tablet_bootstrap.cc:427] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:52.291419 25325 ts_tablet_manager.cc:1173] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.315423 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_6bb13c214301415189d296eaabe6884b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.316143 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.316294 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6bb13c214301415189d296eaabe6884b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.318557 25083 catalog_manager.cc:3663] Got report from deleted tablet acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc): Sending delete request for this tablet
I0619 00:11:52.319223 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acc89522b693498c987ca5d8ef8275ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:52.319321 25082 catalog_manager.cc:3663] Got report from deleted tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.319972 25336 tablet_service.cc:759] Processing CreateTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table=default.table_c3660a004537433aaff9ead44b4615e4 [id=4b052b0654fd4377a294deee53dce6af]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.320055 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.320606 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1dd3f7afe05f49199b18220f9c11ccd2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.323087 25325 tablet_service.cc:800] Processing DeleteTablet for tablet acc89522b693498c987ca5d8ef8275ea with delete_type TABLET_DATA_DELETED (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.324906 25068 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 36 ms (attempt = 2)
W0619 00:11:52.325151 25068 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
W0619 00:11:52.328182 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet c18355df78d344839a56189c7549ab1f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c18355df78d344839a56189c7549ab1f already in progress: creating tablet
I0619 00:11:52.328430 25067 catalog_manager.cc:2891] Scheduling retry of c18355df78d344839a56189c7549ab1f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 128 ms (attempt = 4)
W0619 00:11:52.330335 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 because the tablet was not found. No further retry: Not found: Tablet not found: fc22ad98ed2e4e3eb2c62360626a6f87
W0619 00:11:52.331087 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a4c653a255984d3bac88df63362b9334 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a4c653a255984d3bac88df63362b9334 already in progress: creating tablet
I0619 00:11:52.331338 25067 catalog_manager.cc:2891] Scheduling retry of a4c653a255984d3bac88df63362b9334 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 52 ms (attempt = 1)
I0619 00:11:52.331701 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_43102886b675447088b4319ef7a3b947"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.332573 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_43102886b675447088b4319ef7a3b947 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.311163 25470 ts_tablet_manager.cc:1173] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.340548 25725 ts_tablet_manager.cc:1045] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.094s	user 0.004s	sys 0.004s
I0619 00:11:52.340939 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9d44681acacf4f04b6fcfbd4c9a18e87. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.342188 25725 raft_consensus.cc:325] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.342820 25725 raft_consensus.cc:351] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.343008 25725 raft_consensus.cc:628] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:52.344135 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.345338 25354 raft_consensus.cc:1081] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:52.345515 25819 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3
I0619 00:11:52.348459 25796 consensus_queue.cc:963] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:11:52.353147 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.354358 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_eb0f587538da43c39a0017ca0930cda9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.355286 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_eb0f587538da43c39a0017ca0930cda9 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.357625 25470 tablet_service.cc:759] Processing CreateTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table=default.table_c3660a004537433aaff9ead44b4615e4 [id=4b052b0654fd4377a294deee53dce6af]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.358340 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1dd3f7afe05f49199b18220f9c11ccd2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.359225 25069 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 522 ms (attempt = 6)
W0619 00:11:52.359495 25069 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.298069 25736 ts_tablet_manager.cc:1369] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.362262 25068 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 106 ms (attempt = 3)
W0619 00:11:52.362584 25068 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:52.362725 25727 ts_tablet_manager.cc:1073] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.084s	user 0.006s	sys 0.001s
I0619 00:11:52.364352 25602 ts_tablet_manager.cc:1173] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.347707 25844 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3
I0619 00:11:52.350168 25601 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.351045 25600 tablet_service.cc:759] Processing CreateTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table=default.table_c3660a004537433aaff9ead44b4615e4 [id=4b052b0654fd4377a294deee53dce6af]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.353529 25598 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.260233 25730 ts_tablet_manager.cc:1382] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.365566 25730 log.cc:986] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/aaf62da07a9740c1aa44359d3f604513
I0619 00:11:52.366119 25730 ts_tablet_manager.cc:1403] T aaf62da07a9740c1aa44359d3f604513 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:52.369235 25080 catalog_manager.cc:3663] Got report from deleted tablet acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc): Sending delete request for this tablet
W0619 00:11:52.368216 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.371170 25727 ts_tablet_manager.cc:1028] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:52.371217 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.371879 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acc89522b693498c987ca5d8ef8275ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:52.374250 25067 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC with a delay of 298 ms (attempt = 5)
I0619 00:11:52.374527 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1dd3f7afe05f49199b18220f9c11ccd2. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.374534 25067 catalog_manager.cc:2910] Async tablet task 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 981d862d1bab4b14bb10e6fc2174e1cc consensus state has no leader
W0619 00:11:52.375402 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc22ad98ed2e4e3eb2c62360626a6f87 because the tablet was not found. No further retry: Not found: Tablet not found: fc22ad98ed2e4e3eb2c62360626a6f87
W0619 00:11:52.377027 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a85577b0beba4276b963e5c360544bc7 already in progress: creating tablet
I0619 00:11:52.377274 25067 catalog_manager.cc:2891] Scheduling retry of a85577b0beba4276b963e5c360544bc7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 520 ms (attempt = 6)
I0619 00:11:52.375592 25323 tablet_service.cc:800] Processing DeleteTablet for tablet acc89522b693498c987ca5d8ef8275ea with delete_type TABLET_DATA_DELETED (replaced by tablet 981d862d1bab4b14bb10e6fc2174e1cc) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.377710 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7f60ea5c5def47ee846d0186bbc90c18 already in progress: creating tablet
I0619 00:11:52.378008 25727 tablet_bootstrap.cc:427] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:52.378123 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 78 ms (attempt = 3)
I0619 00:11:52.379333 25724 tablet_bootstrap.cc:578] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:52.380538 25623 raft_consensus.cc:1081] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:52.379231 25741 tablet_replica.cc:261] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:52.381460 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:52.381713 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(4c0614e9ed96420886243a6868f2e75e)
W0619 00:11:52.379550 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.381943 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:52.382174 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:52.382421 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:52.382675 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:52.383078 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(4c0614e9ed96420886243a6868f2e75e)
I0619 00:11:52.383498 25741 raft_consensus.cc:2042] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.383400 25796 consensus_queue.cc:963] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:52.383635 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 107) took 1097ms. Request Metrics: {"tablet-delete.queue_time_us":830917}
I0619 00:11:52.384070 25741 raft_consensus.cc:2069] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.384573 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 109) took 1096ms. Request Metrics: {"tablet-delete.queue_time_us":1096350}
I0619 00:11:52.385449 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 110) took 1097ms. Request Metrics: {"tablet-delete.queue_time_us":1096848}
I0619 00:11:52.386322 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 111) took 1098ms. Request Metrics: {"tablet-delete.queue_time_us":1097671}
I0619 00:11:52.386766 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.387475 25741 ts_tablet_manager.cc:1369] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.388257 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.392925 25336 ts_tablet_manager.cc:1173] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.401298 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 3b80a302aacb418091aee54ab4be7249 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:52.402087 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times.
I0619 00:11:52.405066 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 151 ms (attempt = 4)
W0619 00:11:52.405344 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
W0619 00:11:52.405716 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.408449 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.387747 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.392246 25600 ts_tablet_manager.cc:1173] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.411223 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet aaf62da07a9740c1aa44359d3f604513 (table default.table_48a6321c18954a77805c579d820e8d00 [id=f181d18c1f844c83a0640004333ae473]) successfully deleted
I0619 00:11:52.343431 25725 consensus_queue.cc:227] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.414283 25899 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3
W0619 00:11:52.415534 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet f372b9fb174844eca1662c62039a3c71 because the tablet was not found. No further retry: Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
I0619 00:11:52.416736 25725 ts_tablet_manager.cc:1073] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.076s	user 0.004s	sys 0.000s
W0619 00:11:52.417721 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a because the tablet was not found. No further retry: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
W0619 00:11:52.422369 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.423056 25725 ts_tablet_manager.cc:1028] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:52.423393 25725 tablet_bootstrap.cc:427] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:52.424012 25885 raft_consensus.cc:457] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.424294 25885 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.958s (starting election)
I0619 00:11:52.424480 25727 tablet_bootstrap.cc:578] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.410125 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 113) took 1121ms. Request Metrics: {"tablet-delete.queue_time_us":1120288}
W0619 00:11:52.426211 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a4c653a255984d3bac88df63362b9334 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a4c653a255984d3bac88df63362b9334 already in progress: creating tablet
I0619 00:11:52.428498 25464 ts_tablet_manager.cc:1173] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.430634 25470 ts_tablet_manager.cc:1173] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.424562 25885 raft_consensus.cc:479] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.425846 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 108) took 1138ms. Request Metrics: {"tablet-delete.queue_time_us":1136465}
I0619 00:11:52.433517 25885 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.432126 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 116) took 1032ms. Request Metrics: {"tablet-delete.queue_time_us":1031977}
I0619 00:11:52.435968 25885 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.435525 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 117) took 1035ms. Request Metrics: {"tablet-delete.queue_time_us":1035086}
I0619 00:11:52.437641 25727 tablet_bootstrap.cc:427] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.437901 25730 tablet_replica.cc:261] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:52.438138 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:52.438339 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:52.438526 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:52.437849 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.438751 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:52.438978 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:52.439185 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:52.439384 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:52.439561 25730 raft_consensus.cc:2042] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
W0619 00:11:52.439716 25553 leader_election.cc:306] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.439846 25730 raft_consensus.cc:2069] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.435525 25083 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:52.441185 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.440574 25071 catalog_manager.cc:2891] Scheduling retry of a4c653a255984d3bac88df63362b9334 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 57 ms (attempt = 1)
W0619 00:11:52.441890 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a because the tablet was not found. No further retry: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:52.442497 25081 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:52.433921 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_48d04a9a6f224dd983111f0215fb417b" }
I0619 00:11:52.445837 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 533 ms (attempt = 6)
W0619 00:11:52.446100 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.446477 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 523 ms (attempt = 6)
W0619 00:11:52.446753 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.447202 25067 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 1043 ms (attempt = 7)
W0619 00:11:52.447461 25067 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:52.448367 25080 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_caad097bfc684f938c5e58f0eaa18870 [id=41f48f0e1f87464eb43c5bf2a36f70ef] alter complete (version 3)
I0619 00:11:52.450170 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:52.452442 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.453337 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 03522c48c50943b38cae5aa88562e583 because the tablet was not found. No further retry: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:52.453503 25336 tablet_service.cc:759] Processing CreateTablet for tablet d3c7114381c0423982b9ccbff120c2a2 (table=default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:52.430857 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.453936 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.454330 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.455060 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d3c7114381c0423982b9ccbff120c2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.445495 25725 tablet_bootstrap.cc:578] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:52.460793 25558 leader_election.cc:306] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.461086 25558 leader_election.cc:270] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.461794 25885 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.226s (election complete)
I0619 00:11:52.462018 25885 raft_consensus.cc:2487] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:52.462678 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.456583 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.456804 25081 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:52.463980 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:52.444242 25736 ts_tablet_manager.cc:1382] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.464695 25083 catalog_manager.cc:3663] Got report from deleted tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293): Sending delete request for this tablet
I0619 00:11:52.464824 25736 log.cc:986] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/f372b9fb174844eca1662c62039a3c71
I0619 00:11:52.465353 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
W0619 00:11:52.465458 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.466389 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.466629 25325 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.467097 25736 ts_tablet_manager.cc:1403] T f372b9fb174844eca1662c62039a3c71 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:52.468711 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.471526 25725 tablet_bootstrap.cc:427] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.471900 25725 ts_tablet_manager.cc:1045] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.049s	user 0.005s	sys 0.001s
I0619 00:11:52.472029 25464 tablet_service.cc:759] Processing CreateTablet for tablet d3c7114381c0423982b9ccbff120c2a2 (table=default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.466773 25080 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:52.472836 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.473098 25730 ts_tablet_manager.cc:1369] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.473114 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) successfully deleted
I0619 00:11:52.471289 25068 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 138 ms (attempt = 4)
W0619 00:11:52.473923 25068 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:52.474854 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d3c7114381c0423982b9ccbff120c2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.463943 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.476877 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.477850 25083 catalog_manager.cc:3663] Got report from deleted tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:52.478734 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 35acf55f35064c1491122a13ff9f5744 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.481796 25600 tablet_service.cc:759] Processing CreateTablet for tablet d3c7114381c0423982b9ccbff120c2a2 (table=default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.474859 25725 raft_consensus.cc:325] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.482100 25736 tablet_replica.cc:261] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:52.482282 25725 raft_consensus.cc:351] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.482375 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d3c7114381c0423982b9ccbff120c2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.482360 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.482615 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.482812 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.483006 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.483219 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.483409 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:52.483364 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 522 ms (attempt = 6)
I0619 00:11:52.483611 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(c18355df78d344839a56189c7549ab1f)
W0619 00:11:52.483608 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.483775 25736 raft_consensus.cc:2042] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.483945 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 2088 ms (attempt = 8)
I0619 00:11:52.484014 25736 raft_consensus.cc:2069] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:52.484170 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:52.484503 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 514 ms (attempt = 6)
W0619 00:11:52.484757 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:52.485447 25724 tablet_bootstrap.cc:427] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.482481 25725 raft_consensus.cc:628] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:52.485858 25724 ts_tablet_manager.cc:1045] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.173s	user 0.006s	sys 0.004s
I0619 00:11:52.486582 25725 consensus_queue.cc:227] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.487195 25724 raft_consensus.cc:325] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.487893 25724 raft_consensus.cc:351] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.488112 25724 raft_consensus.cc:628] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.488574 25724 consensus_queue.cc:227] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.489650 25083 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_531af4913e434ddc8afce2aa7b081d1c" }
W0619 00:11:52.489933 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.490932 25470 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:52.491648 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.492036 25724 ts_tablet_manager.cc:1073] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
W0619 00:11:52.492944 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.493252 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.493763 25724 ts_tablet_manager.cc:1028] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.493822 25741 ts_tablet_manager.cc:1382] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.494061 25741 log.cc:986] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/4c0614e9ed96420886243a6868f2e75e
I0619 00:11:52.494238 25724 tablet_bootstrap.cc:427] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:52.494535 25741 ts_tablet_manager.cc:1403] T 4c0614e9ed96420886243a6868f2e75e P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:52.494520 25081 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.494983 25725 ts_tablet_manager.cc:1073] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.023s	user 0.003s	sys 0.001s
I0619 00:11:52.495244 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.496747 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.496990 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 4c0614e9ed96420886243a6868f2e75e (table default.table_581afad997184fb0830cb18cbe8804c3 [id=4a881e07c2634f30bfb601e33cb5d7f0]) successfully deleted
I0619 00:11:52.498585 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3b80a302aacb418091aee54ab4be7249 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.498827 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.499572 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 3b80a302aacb418091aee54ab4be7249 with delete_type TABLET_DATA_DELETED (replaced by tablet acc89522b693498c987ca5d8ef8275ea) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.499953 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0474be659e5e4a4cbfc181b8c812b1eb already in progress: creating tablet
I0619 00:11:52.495823 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 35acf55f35064c1491122a13ff9f5744 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.500654 25071 catalog_manager.cc:2891] Scheduling retry of 0474be659e5e4a4cbfc181b8c812b1eb Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 21 ms (attempt = 1)
I0619 00:11:52.501299 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:52.503974 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 775f0b6b309c49278066a5e7cf623af2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 775f0b6b309c49278066a5e7cf623af2 already in progress: creating tablet
I0619 00:11:52.504616 25071 catalog_manager.cc:2891] Scheduling retry of 775f0b6b309c49278066a5e7cf623af2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 63 ms (attempt = 1)
W0619 00:11:52.505590 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:52.506294 25071 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 80 ms (attempt = 3)
I0619 00:11:52.472277 25727 ts_tablet_manager.cc:1045] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.094s	user 0.005s	sys 0.000s
I0619 00:11:52.508471 25602 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.509042 25336 ts_tablet_manager.cc:1173] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:52.509088 25071 catalog_manager.cc:3249] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): alter failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]),no further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
W0619 00:11:52.510982 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e because the tablet was not found. No further retry: Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
I0619 00:11:52.511023 25081 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_4b9f3e0c22b54ee381137efdf5107650" } new_table_name: "default.table_aa87e3f6cd96435ab36cdbbc83a51374" alter_external_catalogs: true
I0619 00:11:52.512701 25325 tablet_service.cc:759] Processing CreateTablet for tablet fc7686daae274e0dba87703fe6924805 (table=default.table_43102886b675447088b4319ef7a3b947 [id=7bf3a411e3384c2e83d8c19e21acef4d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.513263 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc7686daae274e0dba87703fe6924805. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.513651 25736 ts_tablet_manager.cc:1369] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.517127 25725 ts_tablet_manager.cc:1028] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:52.517513 25725 tablet_bootstrap.cc:427] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:52.517835 25741 tablet_replica.cc:261] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:52.518095 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.518321 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.518556 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.518788 25464 ts_tablet_manager.cc:1173] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.521178 25725 tablet_bootstrap.cc:578] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:52.531175 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 03522c48c50943b38cae5aa88562e583 already in progress: creating tablet
I0619 00:11:52.531460 25067 catalog_manager.cc:2891] Scheduling retry of 03522c48c50943b38cae5aa88562e583 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 295 ms (attempt = 5)
I0619 00:11:52.531936 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
I0619 00:11:52.532604 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.532675 25080 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 47 ms (attempt = 1)
I0619 00:11:52.532817 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.524741 25891 raft_consensus.cc:457] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:52.532937 25080 catalog_manager.cc:3945] Failed to send ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:52.533004 25891 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.207s (starting election)
I0619 00:11:52.533255 25891 raft_consensus.cc:479] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.533717 25885 raft_consensus.cc:457] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.533974 25885 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.732s (starting election)
I0619 00:11:52.534277 25891 leader_election.cc:243] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.534224 25885 raft_consensus.cc:479] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.534687 25891 leader_election.cc:243] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.532402 25323 tablet_service.cc:759] Processing CreateTablet for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table=default.table_6bb13c214301415189d296eaabe6884b [id=088956f856ff4518a56cab0330d74dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.535251 25885 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.535671 25885 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.536767 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "70b9c8c4af0945b69b5577e47b61c054" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.537240 25489 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.850s (vote granted)
I0619 00:11:52.537462 25489 raft_consensus.cc:2245] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:52.537725 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.537714 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:52.538007 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:52.538249 25290 leader_election.cc:306] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054
I0619 00:11:52.538389 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.538637 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.538781 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.538843 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.538712 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7f60ea5c5def47ee846d0186bbc90c18" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.539021 25741 raft_consensus.cc:2042] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.539136 25290 leader_election.cc:379] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.539394 25290 leader_election.cc:270] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:52.539530 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 54adb588955b45b79de9752f254728ea
I0619 00:11:52.539832 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.540316 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.540621 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.541016 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:52.541018 25558 leader_election.cc:306] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.541440 25891 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.160s (election complete)
I0619 00:11:52.541709 25891 raft_consensus.cc:2539] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:52.541810 25081 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 24 ms (attempt = 1)
I0619 00:11:52.541913 25891 raft_consensus.cc:457] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W0619 00:11:52.542106 25081 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.542111 25891 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.628s (starting election)
I0619 00:11:52.542313 25891 raft_consensus.cc:2784] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.545428 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.546448 25081 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
I0619 00:11:52.547226 25081 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 22 ms (attempt = 1)
W0619 00:11:52.547503 25081 catalog_manager.cc:3945] Failed to send ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:52.547859 25600 ts_tablet_manager.cc:1173] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.548005 25891 raft_consensus.cc:479] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.521865 25727 raft_consensus.cc:325] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.548686 25727 raft_consensus.cc:351] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.548880 25727 raft_consensus.cc:628] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.549572 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet 36f5eec972454080b7b434b4df7e62e3 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:52.549305 25727 consensus_queue.cc:227] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.530537 25336 tablet_service.cc:759] Processing CreateTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 (table=default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.550892 25891 leader_election.cc:243] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.551789 25891 leader_election.cc:243] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.552207 25724 tablet_bootstrap.cc:578] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:52.552347 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "70b9c8c4af0945b69b5577e47b61c054" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:52.552790 25741 raft_consensus.cc:2069] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.552795 25489 raft_consensus.cc:2784] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.553038 25489 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.626s (vote granted)
I0619 00:11:52.526937 25325 ts_tablet_manager.cc:1173] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.553683 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7f60ea5c5def47ee846d0186bbc90c18" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.553947 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 71510e2112fd4dc9ba08de08648d7ad7. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.555155 25553 leader_election.cc:306] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.555400 25553 leader_election.cc:270] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.556067 25885 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.816s (election complete)
I0619 00:11:52.556298 25885 raft_consensus.cc:2487] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.557354 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 257 ms (attempt = 5)
W0619 00:11:52.557648 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:52.555959 25464 tablet_service.cc:759] Processing CreateTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 (table=default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:52.559270 25069 catalog_manager.cc:2910] Async tablet task 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a88e394b199431e8b794a1b1aec847f consensus state has no leader
I0619 00:11:52.559509 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b34043f14f1c4edd97a1d4689ced5b32. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.561532 25489 raft_consensus.cc:2245] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:52.562852 25323 ts_tablet_manager.cc:1173] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.564268 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b34043f14f1c4edd97a1d4689ced5b32. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.566049 25290 leader_election.cc:379] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.566252 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 36f5eec972454080b7b434b4df7e62e3
I0619 00:11:52.566336 25290 leader_election.cc:270] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:52.566558 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 36f5eec972454080b7b434b4df7e62e3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:52.566972 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 36f5eec972454080b7b434b4df7e62e3 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet f1b566b2b58f4955aa25ae990a4a8a1a)
W0619 00:11:52.567083 25290 leader_election.cc:306] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054
I0619 00:11:52.568189 25470 tablet_service.cc:759] Processing CreateTablet for tablet fc7686daae274e0dba87703fe6924805 (table=default.table_43102886b675447088b4319ef7a3b947 [id=7bf3a411e3384c2e83d8c19e21acef4d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.568778 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc7686daae274e0dba87703fe6924805. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.570261 25741 ts_tablet_manager.cc:1369] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.574108 25730 ts_tablet_manager.cc:1382] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.574359 25730 log.cc:986] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:52.574836 25730 ts_tablet_manager.cc:1403] T a52cbcbe35374d518bca1e0fc24d88d4 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:52.575052 25725 tablet_bootstrap.cc:427] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.575484 25725 ts_tablet_manager.cc:1045] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.058s	user 0.005s	sys 0.001s
I0619 00:11:52.576189 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 115) took 1176ms. Request Metrics: {"tablet-delete.queue_time_us":995308}
I0619 00:11:52.576982 25470 ts_tablet_manager.cc:1173] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.577628 25807 raft_consensus.cc:2732] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.129s (election complete)
I0619 00:11:52.577865 25807 raft_consensus.cc:2539] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:52.578130 25807 raft_consensus.cc:593] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:52.578524 25807 consensus_queue.cc:204] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.579344 25470 tablet_service.cc:759] Processing CreateTablet for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table=default.table_6bb13c214301415189d296eaabe6884b [id=088956f856ff4518a56cab0330d74dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.580718 25724 tablet_bootstrap.cc:427] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.569867 25068 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 43 ms (attempt = 2)
W0619 00:11:52.581444 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.581475 25724 ts_tablet_manager.cc:1045] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.087s	user 0.004s	sys 0.001s
I0619 00:11:52.579913 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 536 ms (attempt = 6)
W0619 00:11:52.581805 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:52.582324 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 36f5eec972454080b7b434b4df7e62e3 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:52.582381 25727 ts_tablet_manager.cc:1073] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.062s	user 0.003s	sys 0.000s
I0619 00:11:52.583019 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.583688 25727 ts_tablet_manager.cc:1028] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:52.584116 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.583484 25724 raft_consensus.cc:325] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.584533 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 71510e2112fd4dc9ba08de08648d7ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.584544 25724 raft_consensus.cc:351] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.584769 25727 tablet_bootstrap.cc:427] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:52.584772 25724 raft_consensus.cc:628] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.585629 25600 tablet_service.cc:759] Processing CreateTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 (table=default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.585299 25724 consensus_queue.cc:227] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.586256 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b34043f14f1c4edd97a1d4689ced5b32. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.588213 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.588372 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.589675 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.590018 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.590111 25724 ts_tablet_manager.cc:1073] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I0619 00:11:52.590517 25724 ts_tablet_manager.cc:1028] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.590734 25727 tablet_bootstrap.cc:578] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.590924 25724 tablet_bootstrap.cc:427] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:52.594126 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.595933 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 36f5eec972454080b7b434b4df7e62e3 with delete_type TABLET_DATA_DELETED (replaced by tablet f1b566b2b58f4955aa25ae990a4a8a1a) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.596882 25600 ts_tablet_manager.cc:1173] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.591609 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.585377 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.600170 25602 tablet_service.cc:759] Processing CreateTablet for tablet fc7686daae274e0dba87703fe6924805 (table=default.table_43102886b675447088b4319ef7a3b947 [id=7bf3a411e3384c2e83d8c19e21acef4d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.600507 25082 catalog_manager.cc:3663] Got report from deleted tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:52.600752 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc7686daae274e0dba87703fe6924805. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.601181 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.604110 25325 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.604658 25323 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.598300 25083 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]). Expected version 1 got 0
I0619 00:11:52.605649 25083 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 44 ms (attempt = 1)
I0619 00:11:52.605674 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 119) took 1160ms. Request Metrics: {"tablet-delete.queue_time_us":1159629}
I0619 00:11:52.596981 25725 raft_consensus.cc:325] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.606055 25083 catalog_manager.cc:3945] Failed to send 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.607089 25727 tablet_bootstrap.cc:427] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.607396 25727 ts_tablet_manager.cc:1045] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.023s	user 0.003s	sys 0.001s
I0619 00:11:52.608480 25725 raft_consensus.cc:351] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.608729 25725 raft_consensus.cc:628] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:52.609135 25602 ts_tablet_manager.cc:1173] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.610596 25081 catalog_manager.cc:4596] ReplaceTablet: tablet 36f5eec972454080b7b434b4df7e62e3 deleted and replaced by tablet f1b566b2b58f4955aa25ae990a4a8a1a
I0619 00:11:52.610544 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 120) took 1164ms. Request Metrics: {"tablet-delete.queue_time_us":1164306}
I0619 00:11:52.610716 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.611117 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.613168 25068 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 283 ms (attempt = 5)
W0619 00:11:52.613420 25068 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:52.614015 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 121) took 1108ms. Request Metrics: {"tablet-delete.queue_time_us":1107459}
I0619 00:11:52.614554 25730 tablet_replica.cc:261] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:52.614807 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.614998 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.615177 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.615370 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.615599 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.615802 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.615983 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(acc89522b693498c987ca5d8ef8275ea)
I0619 00:11:52.616153 25730 raft_consensus.cc:2042] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.616456 25730 raft_consensus.cc:2069] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.617152 25736 ts_tablet_manager.cc:1382] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.617439 25736 log.cc:986] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/c18355df78d344839a56189c7549ab1f
I0619 00:11:52.617986 25736 ts_tablet_manager.cc:1403] T c18355df78d344839a56189c7549ab1f P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:52.609498 25725 consensus_queue.cc:227] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.620002 25727 raft_consensus.cc:325] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.620625 25727 raft_consensus.cc:351] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.610397 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.620826 25727 raft_consensus.cc:628] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.621261 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 48 ms (attempt = 2)
I0619 00:11:52.621354 25727 consensus_queue.cc:227] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.623435 25727 ts_tablet_manager.cc:1073] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.000s
I0619 00:11:52.625341 25068 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 90 ms (attempt = 3)
W0619 00:11:52.626513 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
W0619 00:11:52.626551 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.626894 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.627348 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 51 ms (attempt = 2)
I0619 00:11:52.628298 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 1035 ms (attempt = 7)
W0619 00:11:52.630053 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
W0619 00:11:52.630105 25067 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:52.632853 25730 ts_tablet_manager.cc:1369] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:52.632879 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.633471 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 163) took 1028ms. Request Metrics: {"tablet-delete.queue_time_us":875819}
I0619 00:11:52.633718 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c3660a004537433aaff9ead44b4615e4" } new_table_name: "default.table_c24f1b02a5b54618aca5028342d84b7d" alter_external_catalogs: true
I0619 00:11:52.625811 25336 ts_tablet_manager.cc:1173] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:52.627885 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.627869 25807 raft_consensus.cc:457] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.634528 25807 raft_consensus.cc:2732] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.661s (starting election)
I0619 00:11:52.634798 25807 raft_consensus.cc:479] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.635711 25807 leader_election.cc:243] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.636090 25807 leader_election.cc:243] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:52.637945 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.638968 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) successfully deleted
W0619 00:11:52.640880 25290 leader_election.cc:306] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
W0619 00:11:52.641371 25290 leader_election.cc:306] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
I0619 00:11:52.641650 25290 leader_election.cc:270] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.642702 25807 raft_consensus.cc:2732] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.735s (election complete)
I0619 00:11:52.642953 25807 raft_consensus.cc:2487] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.640275 25847 raft_consensus.cc:457] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:52.641957 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.643798 25082 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC with a delay of 50 ms (attempt = 1)
I0619 00:11:52.643971 25847 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.159s (starting election)
W0619 00:11:52.644071 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
I0619 00:11:52.644235 25847 raft_consensus.cc:479] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.645249 25847 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.645491 25727 ts_tablet_manager.cc:1028] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:52.645666 25847 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.645642 25083 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:52.645874 25727 tablet_bootstrap.cc:427] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:52.646898 25083 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 52 ms (attempt = 1)
W0619 00:11:52.647192 25083 catalog_manager.cc:3945] Failed to send 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.647231 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:52.648484 25419 leader_election.cc:306] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.649590 25727 tablet_bootstrap.cc:578] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.650333 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c644f13df4b147a08114a4e3d266c0d4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.651279 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c644f13df4b147a08114a4e3d266c0d4 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.651214 25082 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:52.652253 25082 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 22 ms (attempt = 1)
W0619 00:11:52.652534 25082 catalog_manager.cc:3945] Failed to send 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.653126 25807 raft_consensus.cc:457] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.653224 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 36f5eec972454080b7b434b4df7e62e3 with delete_type TABLET_DATA_DELETED (replaced by tablet f1b566b2b58f4955aa25ae990a4a8a1a) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.653383 25807 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.872s (starting election)
I0619 00:11:52.653594 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.653713 25807 raft_consensus.cc:479] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.643359 25725 ts_tablet_manager.cc:1073] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.068s	user 0.005s	sys 0.000s
I0619 00:11:52.654516 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.654747 25725 ts_tablet_manager.cc:1028] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:52.654830 25807 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.617813 25724 tablet_bootstrap.cc:578] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:52.650796 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:52.655133 25725 tablet_bootstrap.cc:427] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:52.655563 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 36f5eec972454080b7b434b4df7e62e3 with delete_type TABLET_DATA_DELETED (replaced by tablet f1b566b2b58f4955aa25ae990a4a8a1a) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.655755 25623 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.965s (vote granted)
I0619 00:11:52.656002 25623 raft_consensus.cc:2245] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:52.657160 25423 leader_election.cc:379] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.657400 25423 leader_election.cc:270] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:52.657840 25725 tablet_bootstrap.cc:578] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:52.659202 25847 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.216s (election complete)
I0619 00:11:52.659427 25847 raft_consensus.cc:2539] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:52.659646 25847 raft_consensus.cc:457] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:52.659833 25847 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.122s (starting election)
I0619 00:11:52.660002 25847 raft_consensus.cc:2784] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.660286 25068 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 79 ms (attempt = 2)
I0619 00:11:52.660496 25464 ts_tablet_manager.cc:1173] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:52.660557 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.661837 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b2764b59c43146a8ae54b361c93b3406"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.662896 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b2764b59c43146a8ae54b361c93b3406 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.664708 25602 tablet_service.cc:759] Processing CreateTablet for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table=default.table_6bb13c214301415189d296eaabe6884b [id=088956f856ff4518a56cab0330d74dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.665030 25847 raft_consensus.cc:479] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.665851 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 71510e2112fd4dc9ba08de08648d7ad7. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.666009 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.666261 25847 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.666963 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) successfully deleted
I0619 00:11:52.667747 25847 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.667870 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:52.669371 25419 leader_election.cc:306] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.670409 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:52.670895 25623 raft_consensus.cc:2784] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.671135 25623 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.758s (vote granted)
I0619 00:11:52.671033 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_673676f1e1e94d76afe2c9b06d701801"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.671900 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_673676f1e1e94d76afe2c9b06d701801 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.672821 25081 catalog_manager.cc:3663] Got report from deleted tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (replaced by tablet a85577b0beba4276b963e5c360544bc7): Sending delete request for this tablet
W0619 00:11:52.673892 25067 catalog_manager.cc:2910] Async tablet task 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 981d862d1bab4b14bb10e6fc2174e1cc consensus state has no leader
I0619 00:11:52.675490 25623 raft_consensus.cc:2245] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:52.676635 25423 leader_election.cc:379] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.676870 25423 leader_election.cc:270] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:52.677018 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 70b9c8c4af0945b69b5577e47b61c054 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet a85577b0beba4276b963e5c360544bc7)
I0619 00:11:52.677347 25796 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.630s (election complete)
I0619 00:11:52.677575 25796 raft_consensus.cc:2539] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:52.677896 25796 raft_consensus.cc:593] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:52.678333 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_2af4c8dc619643729ba189a0dd8cd15f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:52.678323 25796 consensus_queue.cc:204] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.679198 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2af4c8dc619643729ba189a0dd8cd15f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:52.679800 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.682440 25323 tablet_service.cc:759] Processing CreateTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a (table=default.table_e4aaded22fbb4c60a591e5ff45bfe2bc [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.683075 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f1b566b2b58f4955aa25ae990a4a8a1a. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.683642 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.684543 25082 catalog_manager.cc:3663] Got report from deleted tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b): Sending delete request for this tablet
I0619 00:11:52.631515 25741 ts_tablet_manager.cc:1382] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.684964 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 48 ms (attempt = 2)
W0619 00:11:52.685225 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.685035 25741 log.cc:986] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/acc89522b693498c987ca5d8ef8275ea
I0619 00:11:52.685919 25741 ts_tablet_manager.cc:1403] T acc89522b693498c987ca5d8ef8275ea P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:52.691352 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0474be659e5e4a4cbfc181b8c812b1eb already in progress: creating tablet
I0619 00:11:52.691651 25067 catalog_manager.cc:2891] Scheduling retry of 0474be659e5e4a4cbfc181b8c812b1eb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 26 ms (attempt = 1)
I0619 00:11:52.692214 25725 tablet_bootstrap.cc:427] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.692572 25725 ts_tablet_manager.cc:1045] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.038s	user 0.003s	sys 0.001s
I0619 00:11:52.685232 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f372b9fb174844eca1662c62039a3c71 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 30a9d9fda8d34caeb8242c372439321b)
W0619 00:11:52.695839 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:52.695864 25807 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.696822 25724 tablet_bootstrap.cc:427] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.697141 25724 ts_tablet_manager.cc:1045] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.106s	user 0.003s	sys 0.001s
I0619 00:11:52.697917 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 103 ms (attempt = 3)
I0619 00:11:52.698441 25724 raft_consensus.cc:325] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.699069 25724 raft_consensus.cc:351] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.699250 25724 raft_consensus.cc:628] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.699893 25068 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC with a delay of 52 ms (attempt = 2)
W0619 00:11:52.699307 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
W0619 00:11:52.700206 25068 catalog_manager.cc:2910] Async tablet task 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
I0619 00:11:52.699774 25724 consensus_queue.cc:227] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.700619 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 81 ms (attempt = 3)
I0619 00:11:52.700949 25325 tablet_service.cc:800] Processing DeleteTablet for tablet f372b9fb174844eca1662c62039a3c71 with delete_type TABLET_DATA_DELETED (replaced by tablet 30a9d9fda8d34caeb8242c372439321b) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.704408 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.702306 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 775f0b6b309c49278066a5e7cf623af2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 775f0b6b309c49278066a5e7cf623af2 already in progress: creating tablet
I0619 00:11:52.704674 25067 catalog_manager.cc:2891] Scheduling retry of 775f0b6b309c49278066a5e7cf623af2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 52 ms (attempt = 1)
W0619 00:11:52.705126 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
W0619 00:11:52.706004 25290 leader_election.cc:306] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:52.706295 25464 tablet_service.cc:759] Processing CreateTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a (table=default.table_e4aaded22fbb4c60a591e5ff45bfe2bc [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.707427 25600 tablet_service.cc:759] Processing CreateTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a (table=default.table_e4aaded22fbb4c60a591e5ff45bfe2bc [id=122dad4ddc134d21b827dca8587ed257]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:52.707796 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.708032 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f1b566b2b58f4955aa25ae990a4a8a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.708695 25323 ts_tablet_manager.cc:1173] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.665484 25470 ts_tablet_manager.cc:1173] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.711267 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb57d196f0724247b6a7f9a34be8377a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.711733 25489 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.066s (vote granted)
I0619 00:11:52.711964 25489 raft_consensus.cc:2245] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:52.713230 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 166) took 1063ms. Request Metrics: {"tablet-delete.queue_time_us":1015770}
I0619 00:11:52.714331 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 70b9c8c4af0945b69b5577e47b61c054 with delete_type TABLET_DATA_DELETED (replaced by tablet a85577b0beba4276b963e5c360544bc7) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.714565 25290 leader_election.cc:379] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.714828 25290 leader_election.cc:270] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:52.715253 25807 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.167s (election complete)
I0619 00:11:52.715951 25807 raft_consensus.cc:2539] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:52.716172 25807 raft_consensus.cc:457] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:52.716382 25807 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.163s (starting election)
I0619 00:11:52.716614 25807 raft_consensus.cc:2784] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.717124 25068 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 164 ms (attempt = 4)
I0619 00:11:52.717564 25600 ts_tablet_manager.cc:1173] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:52.717803 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.718458 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.720655 25807 raft_consensus.cc:479] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.719743 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 40 ms (attempt = 2)
I0619 00:11:52.721524 25807 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.721906 25807 leader_election.cc:243] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.722189 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.723230 25724 ts_tablet_manager.cc:1073] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.001s
I0619 00:11:52.723248 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb57d196f0724247b6a7f9a34be8377a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:52.723675 25724 ts_tablet_manager.cc:1028] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.723703 25489 raft_consensus.cc:2784] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.723955 25489 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.939s (vote granted)
I0619 00:11:52.724051 25724 tablet_bootstrap.cc:427] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:52.724125 25736 tablet_replica.cc:261] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:52.724386 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:52.724591 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:52.724807 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:52.724997 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:52.725215 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:52.725380 25727 tablet_bootstrap.cc:427] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.725417 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:52.725597 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:52.725796 25736 raft_consensus.cc:2042] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.726078 25727 ts_tablet_manager.cc:1045] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.080s	user 0.004s	sys 0.001s
I0619 00:11:52.726056 25736 raft_consensus.cc:2069] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.727279 25724 tablet_bootstrap.cc:578] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:52.727356 25736 ts_tablet_manager.cc:1369] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.727442 25727 raft_consensus.cc:325] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.728080 25727 raft_consensus.cc:351] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.728292 25727 raft_consensus.cc:628] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.728580 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f1b566b2b58f4955aa25ae990a4a8a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.728770 25727 consensus_queue.cc:227] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.729557 25489 raft_consensus.cc:2245] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
W0619 00:11:52.731205 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.731405 25727 ts_tablet_manager.cc:1073] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:11:52.731771 25727 ts_tablet_manager.cc:1028] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:52.731940 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 53688f0e4a604bc4a07dec9278a8dad6 because the tablet was not found. No further retry: Not found: Tablet not found: 53688f0e4a604bc4a07dec9278a8dad6
I0619 00:11:52.732123 25727 tablet_bootstrap.cc:427] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:52.732470 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet aaf62da07a9740c1aa44359d3f604513 because the tablet was not found. No further retry: Not found: Tablet not found: aaf62da07a9740c1aa44359d3f604513
W0619 00:11:52.733014 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4c0614e9ed96420886243a6868f2e75e because the tablet was not found. No further retry: Not found: Tablet not found: 4c0614e9ed96420886243a6868f2e75e
I0619 00:11:52.733541 25290 leader_election.cc:379] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:52.733633 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 54373917567a479d93dc8a3fd006d2fe with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 54373917567a479d93dc8a3fd006d2fe already in progress: creating tablet
I0619 00:11:52.733810 25290 leader_election.cc:270] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:52.734009 25067 catalog_manager.cc:2891] Scheduling retry of 54373917567a479d93dc8a3fd006d2fe Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 33 ms (attempt = 1)
I0619 00:11:52.734243 25891 raft_consensus.cc:2732] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.544s (election complete)
I0619 00:11:52.734467 25891 raft_consensus.cc:2539] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
W0619 00:11:52.734788 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 because the tablet was not found. No further retry: Not found: Tablet not found: 292b164659e343158de8e7e3eeb9be31
W0619 00:11:52.735312 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 292b164659e343158de8e7e3eeb9be31 because the tablet was not found. No further retry: Not found: Tablet not found: 292b164659e343158de8e7e3eeb9be31
W0619 00:11:52.735801 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 397ae10bb6d841b4ba561ec431d29864 already in progress: creating tablet
I0619 00:11:52.736044 25067 catalog_manager.cc:2891] Scheduling retry of 397ae10bb6d841b4ba561ec431d29864 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 539 ms (attempt = 6)
I0619 00:11:52.738111 25891 raft_consensus.cc:593] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:52.738639 25081 catalog_manager.cc:3663] Got report from deleted tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.738526 25891 consensus_queue.cc:204] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.739856 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.739370 25725 raft_consensus.cc:325] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.740195 25725 raft_consensus.cc:351] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.740413 25725 raft_consensus.cc:628] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:52.740887 25068 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 80 ms (attempt = 3)
W0619 00:11:52.741140 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.741068 25725 consensus_queue.cc:227] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.741605 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.746337 25741 tablet_replica.cc:261] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:52.746637 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.746866 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.747081 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.747297 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.747542 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.747787 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.748004 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(f372b9fb174844eca1662c62039a3c71)
I0619 00:11:52.748204 25741 raft_consensus.cc:2042] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.748481 25741 raft_consensus.cc:2069] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:52.751302 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.721822 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.752789 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet d6c3c7f783cb43d59b46502261aeced3 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:52.754000 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.750094 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.750484 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.752821 25290 leader_election.cc:306] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:52.754390 25068 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC with a delay of 100 ms (attempt = 3)
W0619 00:11:52.754710 25068 catalog_manager.cc:2910] Async tablet task 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
I0619 00:11:52.755098 25807 raft_consensus.cc:457] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.755313 25807 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.192s (starting election)
I0619 00:11:52.755569 25807 raft_consensus.cc:479] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.756961 25725 ts_tablet_manager.cc:1073] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.062s	user 0.004s	sys 0.000s
I0619 00:11:52.756548 25807 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.757558 25807 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.757510 25725 ts_tablet_manager.cc:1028] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:52.758086 25725 tablet_bootstrap.cc:427] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:52.757524 25080 catalog_manager.cc:3786] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:52.758613 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:52.760521 25725 tablet_bootstrap.cc:578] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:52.762344 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:52.763782 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:52.763846 25741 ts_tablet_manager.cc:1369] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.764580 25071 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC with a delay of 99 ms (attempt = 3)
I0619 00:11:52.766309 25730 ts_tablet_manager.cc:1382] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.767824 25323 tablet_service.cc:759] Processing CreateTablet for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table=default.table_c644f13df4b147a08114a4e3d266c0d4 [id=e3efb205417f4b40b4f48094be7d1e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.768452 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 41e052fdd92b4c3d9803fa43433e60d5. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:52.769312 25071 catalog_manager.cc:2910] Async tablet task 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 3722b203a516466489c74ef6013eacae consensus state has no leader
I0619 00:11:52.771438 25336 tablet_service.cc:759] Processing CreateTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 (table=default.table_b2764b59c43146a8ae54b361c93b3406 [id=464afb6b12f4446b89a9682f1d407e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.768645 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.772078 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e60ecc51c92c461889b360dde1dc5cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.772814 25068 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 2091 ms (attempt = 8)
W0619 00:11:52.773088 25068 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
W0619 00:11:52.773815 25290 leader_election.cc:306] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:52.774188 25080 catalog_manager.cc:3663] Got report from deleted tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:52.774705 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.774844 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.776338 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.778893 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:52.779579 25080 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 53 ms (attempt = 1)
W0619 00:11:52.779952 25080 catalog_manager.cc:3945] Failed to send 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
W0619 00:11:52.781461 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.781713 25730 log.cc:986] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/acc89522b693498c987ca5d8ef8275ea
I0619 00:11:52.782207 25730 ts_tablet_manager.cc:1403] T acc89522b693498c987ca5d8ef8275ea P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:52.773438 25724 tablet_bootstrap.cc:427] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.782846 25724 ts_tablet_manager.cc:1045] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.059s	user 0.004s	sys 0.001s
I0619 00:11:52.784533 25724 raft_consensus.cc:325] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.785220 25724 raft_consensus.cc:351] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.785804 25724 raft_consensus.cc:628] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.785935 25336 ts_tablet_manager.cc:1173] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.747632 25727 tablet_bootstrap.cc:578] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.786398 25724 consensus_queue.cc:227] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.770979 25602 ts_tablet_manager.cc:1173] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.763195 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03522c48c50943b38cae5aa88562e583" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.791265 25489 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.005s (vote granted)
I0619 00:11:52.791504 25489 raft_consensus.cc:2245] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:52.792011 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet d6c3c7f783cb43d59b46502261aeced3
I0619 00:11:52.794142 25470 tablet_service.cc:759] Processing CreateTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 (table=default.table_b2764b59c43146a8ae54b361c93b3406 [id=464afb6b12f4446b89a9682f1d407e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.794584 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:52.794775 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e60ecc51c92c461889b360dde1dc5cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.794751 25468 tablet_service.cc:759] Processing CreateTablet for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table=default.table_c644f13df4b147a08114a4e3d266c0d4 [id=e3efb205417f4b40b4f48094be7d1e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.795218 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:52.795352 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 41e052fdd92b4c3d9803fa43433e60d5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.795614 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:52.795835 25323 ts_tablet_manager.cc:1173] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.796010 25081 catalog_manager.cc:4596] ReplaceTablet: tablet d6c3c7f783cb43d59b46502261aeced3 deleted and replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d
I0619 00:11:52.797621 25602 tablet_service.cc:759] Processing CreateTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 (table=default.table_b2764b59c43146a8ae54b361c93b3406 [id=464afb6b12f4446b89a9682f1d407e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.798252 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e60ecc51c92c461889b360dde1dc5cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.798254 25736 ts_tablet_manager.cc:1382] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:11:52.799238 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.800351 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 123) took 1293ms. Request Metrics: {"mutex_wait_us":9583,"tablet-delete.queue_time_us":1107339}
I0619 00:11:52.801331 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 124) took 1258ms. Request Metrics: {"tablet-delete.queue_time_us":1257265}
I0619 00:11:52.802266 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 125) took 1245ms. Request Metrics: {"tablet-delete.queue_time_us":1245082}
I0619 00:11:52.803146 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 127) took 1242ms. Request Metrics: {"tablet-delete.queue_time_us":1241716}
I0619 00:11:52.804018 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 126) took 1244ms. Request Metrics: {"tablet-delete.queue_time_us":1241909}
I0619 00:11:52.805222 25730 tablet_replica.cc:261] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:52.805354 25600 tablet_service.cc:759] Processing CreateTablet for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table=default.table_c644f13df4b147a08114a4e3d266c0d4 [id=e3efb205417f4b40b4f48094be7d1e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.805850 25736 log.cc:986] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/70b9c8c4af0945b69b5577e47b61c054
I0619 00:11:52.805945 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 41e052fdd92b4c3d9803fa43433e60d5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.806326 25736 ts_tablet_manager.cc:1403] T 70b9c8c4af0945b69b5577e47b61c054 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
W0619 00:11:52.808652 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.810407 25081 catalog_manager.cc:3663] Got report from deleted tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:52.811141 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.813642 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 147 ms (attempt = 4)
I0619 00:11:52.814486 25464 ts_tablet_manager.cc:1173] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.814846 25724 ts_tablet_manager.cc:1073] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.032s	user 0.004s	sys 0.000s
I0619 00:11:52.815249 25724 ts_tablet_manager.cc:1028] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.815670 25724 tablet_bootstrap.cc:427] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:52.815593 25081 catalog_manager.cc:3663] Got report from deleted tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:52.815815 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 529 ms (attempt = 6)
W0619 00:11:52.816109 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:52.819680 25623 raft_consensus.cc:1081] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:52.821096 25847 consensus_queue.cc:963] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:52.821869 25068 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 140 ms (attempt = 4)
W0619 00:11:52.822137 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.825856 25468 ts_tablet_manager.cc:1173] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:52.826634 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.827097 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.828393 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.831765 25470 ts_tablet_manager.cc:1173] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.797224 25290 leader_election.cc:379] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.799809 25336 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.833784 25290 leader_election.cc:270] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:52.833905 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 78 ms (attempt = 2)
W0619 00:11:52.834175 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.834458 25891 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.762s (election complete)
I0619 00:11:52.834718 25891 raft_consensus.cc:2539] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W0619 00:11:52.813930 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:52.834926 25891 raft_consensus.cc:457] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:52.835054 25727 tablet_bootstrap.cc:427] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.835289 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.835433 25727 ts_tablet_manager.cc:1045] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.103s	user 0.005s	sys 0.001s
I0619 00:11:52.835984 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 162 ms (attempt = 4)
W0619 00:11:52.836033 25419 consensus_peers.cc:434] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 3722b203a516466489c74ef6013eacae. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:52.835116 25891 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.801s (starting election)
I0619 00:11:52.836555 25891 raft_consensus.cc:2784] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.837405 25727 raft_consensus.cc:325] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.838017 25727 raft_consensus.cc:351] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.838196 25727 raft_consensus.cc:628] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.839473 25727 consensus_queue.cc:227] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.839995 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:52.840221 25336 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.841543 25602 ts_tablet_manager.cc:1173] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.841043 25891 raft_consensus.cc:479] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.842993 25891 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.843391 25891 leader_election.cc:243] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:52.844600 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 72e28bc6b57442b782d8aa07e1152cf4 because the tablet was not found. No further retry: Not found: Tablet not found: 72e28bc6b57442b782d8aa07e1152cf4
I0619 00:11:52.844858 25470 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.846266 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 167) took 1196ms. Request Metrics: {"tablet-delete.queue_time_us":1045973}
I0619 00:11:52.847329 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 171) took 1016ms. Request Metrics: {"tablet-delete.queue_time_us":1015337}
I0619 00:11:52.805491 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:52.849321 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03522c48c50943b38cae5aa88562e583" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:52.849776 25489 raft_consensus.cc:2784] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.850019 25489 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.648s (vote granted)
W0619 00:11:52.851137 25290 leader_election.cc:306] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:52.853281 25600 ts_tablet_manager.cc:1173] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:52.851511 25727 ts_tablet_manager.cc:1073] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.000s
W0619 00:11:52.851475 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.856127 25727 ts_tablet_manager.cc:1028] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:52.855871 25068 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC with a delay of 168 ms (attempt = 4)
W0619 00:11:52.857870 25068 catalog_manager.cc:2910] Async tablet task 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
I0619 00:11:52.859033 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.859503 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:52.859539 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:52.860008 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:52.857877 25727 tablet_bootstrap.cc:427] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:52.860085 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.860740 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:52.857134 25081 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:52.861444 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.861549 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.861265 25724 tablet_bootstrap.cc:578] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:52.864239 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0474be659e5e4a4cbfc181b8c812b1eb already in progress: creating tablet
I0619 00:11:52.864464 25071 catalog_manager.cc:2891] Scheduling retry of 0474be659e5e4a4cbfc181b8c812b1eb Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 33 ms (attempt = 1)
I0619 00:11:52.864835 25727 tablet_bootstrap.cc:578] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.865211 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.861465 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:52.866688 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:52.866915 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:52.867120 25730 raft_consensus.cc:2042] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.867416 25730 raft_consensus.cc:2069] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:52.869078 25730 ts_tablet_manager.cc:1369] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.870090 25489 raft_consensus.cc:2245] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:52.848392 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 172) took 1012ms. Request Metrics: {"tablet-delete.queue_time_us":1011573}
I0619 00:11:52.871692 25290 leader_election.cc:379] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.872086 25290 leader_election.cc:270] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:52.872120 25082 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:52.872174 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_cc45383880634c41948247ee9fa098d1" }
I0619 00:11:52.872853 25081 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_aa87e3f6cd96435ab36cdbbc83a51374" }
I0619 00:11:52.873092 25891 raft_consensus.cc:2732] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.024s (election complete)
I0619 00:11:52.873180 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.873461 25891 raft_consensus.cc:2539] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:52.875280 25600 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.874050 25891 raft_consensus.cc:593] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
W0619 00:11:52.876446 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.876521 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:52.874202 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet c18355df78d344839a56189c7549ab1f because the tablet was not found. No further retry: Not found: Tablet not found: c18355df78d344839a56189c7549ab1f
I0619 00:11:52.875979 25891 consensus_queue.cc:204] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.878496 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_e4aaded22fbb4c60a591e5ff45bfe2bc" } new_table_name: "default.table_970d5da055774f58881ea62e91e2f1f4" alter_external_catalogs: true
I0619 00:11:52.879065 25741 ts_tablet_manager.cc:1382] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:52.879300 25741 log.cc:986] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/f372b9fb174844eca1662c62039a3c71
I0619 00:11:52.880288 25725 tablet_bootstrap.cc:427] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.880621 25725 ts_tablet_manager.cc:1045] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.123s	user 0.004s	sys 0.003s
I0619 00:11:52.880749 25736 tablet_replica.cc:261] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:52.881109 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:52.881458 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:52.881752 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:52.882046 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:52.881959 25725 raft_consensus.cc:325] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.882452 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:52.882513 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) successfully deleted
I0619 00:11:52.882808 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:52.883209 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:52.883491 25736 raft_consensus.cc:2042] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:52.883896 25736 raft_consensus.cc:2069] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:52.884529 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet c18355df78d344839a56189c7549ab1f because the tablet was not found. No further retry: Not found: Tablet not found: c18355df78d344839a56189c7549ab1f
I0619 00:11:52.887188 25068 catalog_manager.cc:2891] Scheduling retry of 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC with a delay of 301 ms (attempt = 5)
W0619 00:11:52.887470 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.891661 25724 tablet_bootstrap.cc:427] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:52.891762 25323 tablet_service.cc:759] Processing CreateTablet for tablet 1cfd335c18c047fba2facc5194807abb (table=default.table_2af4c8dc619643729ba189a0dd8cd15f [id=ad26a268092e4cc4a44e6845dc511880]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.892076 25724 ts_tablet_manager.cc:1045] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.077s	user 0.004s	sys 0.000s
I0619 00:11:52.892359 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1cfd335c18c047fba2facc5194807abb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.893762 25082 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 40 ms (attempt = 1)
W0619 00:11:52.894042 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:52.897899 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:52.898394 25068 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC with a delay of 543 ms (attempt = 6)
W0619 00:11:52.898676 25068 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:52.898857 25336 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.899150 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c644f13df4b147a08114a4e3d266c0d4" } new_table_name: "default.table_34ae54d8d2fc4703a6fd963ddd934b4d" alter_external_catalogs: true
I0619 00:11:52.882602 25725 raft_consensus.cc:351] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.900781 25725 raft_consensus.cc:628] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:52.901232 25083 catalog_manager.cc:3663] Got report from deleted tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.901824 25736 ts_tablet_manager.cc:1369] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:52.901886 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.901386 25725 consensus_queue.cc:227] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.902684 25899 tablet.cc:1200] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
W0619 00:11:52.904444 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.904022 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet aaf62da07a9740c1aa44359d3f604513 because the tablet was not found. No further retry: Not found: Tablet not found: aaf62da07a9740c1aa44359d3f604513
I0619 00:11:52.905737 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:52.907436 25725 ts_tablet_manager.cc:1073] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.000s
I0619 00:11:52.907832 25725 ts_tablet_manager.cc:1028] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:52.908182 25725 tablet_bootstrap.cc:427] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:52.908298 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:52.909713 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.911274 25725 tablet_bootstrap.cc:578] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:52.913331 25083 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:52.913918 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 66 ms (attempt = 3)
W0619 00:11:52.914173 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.915781 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet d62715e230154e2abe126ef97feec7ce
I0619 00:11:52.916079 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.916445 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.916823 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.917852 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.918256 25325 tablet_service.cc:800] Processing DeleteTablet for tablet d62715e230154e2abe126ef97feec7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.919236 25083 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:52.919450 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:52.919899 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.928210 25083 master_service.cc:523] ReplaceTablet: received request to replace tablet f372b9fb174844eca1662c62039a3c71 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:52.926739 25923 tablet.cc:1200] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:52.928762 25325 tablet_service.cc:759] Processing CreateTablet for tablet 6040e369e0fb445fb993d12dd883039b (table=default.table_673676f1e1e94d76afe2c9b06d701801 [id=980403da36b04fad89427ba22b96d098]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:52.929203 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7f60ea5c5def47ee846d0186bbc90c18 already in progress: creating tablet
I0619 00:11:52.915513 25724 raft_consensus.cc:325] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.931169 25724 raft_consensus.cc:351] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.930359 25071 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 40 ms (attempt = 2)
I0619 00:11:52.931362 25724 raft_consensus.cc:628] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:52.933694 25724 consensus_queue.cc:227] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.934917 25069 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 62 ms (attempt = 2)
W0619 00:11:52.935184 25069 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:52.932189 25775 raft_consensus.cc:457] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.935823 25775 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.977s (starting election)
I0619 00:11:52.936110 25775 raft_consensus.cc:479] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.937126 25775 leader_election.cc:243] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.937520 25775 leader_election.cc:243] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:52.939272 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.940623 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee4c4769f75141f9b2ef864ab8a8d9e0" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:52.941061 25623 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.135s (vote granted)
I0619 00:11:52.941277 25623 raft_consensus.cc:2245] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:52.942737 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee4c4769f75141f9b2ef864ab8a8d9e0" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:52.944183 25419 leader_election.cc:306] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.945546 25082 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 17 ms (attempt = 1)
I0619 00:11:52.945610 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 7c57c5ca55f24621b66197a3698504bd
W0619 00:11:52.945866 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:52.945961 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.946398 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.946786 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:52.947052 25423 leader_election.cc:379] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.947290 25423 leader_election.cc:270] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:52.947873 25775 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.871s (election complete)
I0619 00:11:52.947916 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:52.948102 25775 raft_consensus.cc:2539] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:52.948307 25775 raft_consensus.cc:457] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:52.948508 25775 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.990s (starting election)
I0619 00:11:52.948735 25775 raft_consensus.cc:2784] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:52.951653 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_81b1d81c6c2b4710915f7ea29e3ec00b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:52.952267 25885 raft_consensus.cc:457] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.952509 25885 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.013s (starting election)
I0619 00:11:52.952771 25885 raft_consensus.cc:479] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.953759 25885 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.953764 25775 raft_consensus.cc:479] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.954195 25885 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:52.954699 25775 leader_election.cc:243] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:52.955500 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_81b1d81c6c2b4710915f7ea29e3ec00b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.955965 25775 leader_election.cc:243] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.955821 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:52.957046 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee4c4769f75141f9b2ef864ab8a8d9e0" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:52.957363 25553 leader_election.cc:306] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.960863 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:52.961014 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee4c4769f75141f9b2ef864ab8a8d9e0" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:52.961486 25623 raft_consensus.cc:2784] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:52.961524 25419 leader_election.cc:306] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.961752 25623 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.957s (vote granted)
I0619 00:11:52.964845 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.966235 25600 tablet_service.cc:800] Processing DeleteTablet for tablet d62715e230154e2abe126ef97feec7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:52.966817 25558 leader_election.cc:306] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.967082 25558 leader_election.cc:270] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:52.967525 25885 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.872s (election complete)
I0619 00:11:52.967772 25885 raft_consensus.cc:2487] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:52.967243 25600 tablet_service.cc:759] Processing CreateTablet for tablet 1cfd335c18c047fba2facc5194807abb (table=default.table_2af4c8dc619643729ba189a0dd8cd15f [id=ad26a268092e4cc4a44e6845dc511880]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:52.968068 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:52.968461 25725 tablet_bootstrap.cc:427] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:52.968436 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 74 ms (attempt = 2)
W0619 00:11:52.968734 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:52.968844 25725 ts_tablet_manager.cc:1045] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.061s	user 0.005s	sys 0.000s
I0619 00:11:52.969431 25602 tablet_service.cc:759] Processing CreateTablet for tablet 6040e369e0fb445fb993d12dd883039b (table=default.table_673676f1e1e94d76afe2c9b06d701801 [id=980403da36b04fad89427ba22b96d098]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.970444 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:52.971254 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 1064 ms (attempt = 7)
W0619 00:11:52.971524 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.970731 25725 raft_consensus.cc:325] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.972347 25725 raft_consensus.cc:351] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:52.972535 25725 raft_consensus.cc:628] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:52.972359 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3f22e2fb4a9e47aca16e414746eecd63"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.973269 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3f22e2fb4a9e47aca16e414746eecd63 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.973058 25725 consensus_queue.cc:227] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:52.977301 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.979349 25470 tablet_service.cc:800] Processing DeleteTablet for tablet d62715e230154e2abe126ef97feec7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.981146 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.981258 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 1071 ms (attempt = 7)
W0619 00:11:52.981945 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.982033 25470 tablet_service.cc:759] Processing CreateTablet for tablet 1cfd335c18c047fba2facc5194807abb (table=default.table_2af4c8dc619643729ba189a0dd8cd15f [id=ad26a268092e4cc4a44e6845dc511880]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.982250 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 139 ms (attempt = 4)
W0619 00:11:52.982466 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:52.981536 25724 ts_tablet_manager.cc:1073] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.088s	user 0.003s	sys 0.000s
I0619 00:11:52.982848 25724 ts_tablet_manager.cc:1028] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:52.977252 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:52.983407 25468 tablet_service.cc:759] Processing CreateTablet for tablet 6040e369e0fb445fb993d12dd883039b (table=default.table_673676f1e1e94d76afe2c9b06d701801 [id=980403da36b04fad89427ba22b96d098]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:52.984035 25724 tablet_bootstrap.cc:427] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:52.984076 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:52.984264 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_559173a19fea4dfea574c9469397df05"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:52.985139 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_559173a19fea4dfea574c9469397df05 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:52.984109 25775 raft_consensus.cc:457] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:52.992002 25775 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.570s (starting election)
I0619 00:11:52.992265 25775 raft_consensus.cc:479] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.993248 25775 leader_election.cc:243] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:52.993484 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 282 ms (attempt = 5)
I0619 00:11:52.994824 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "397ae10bb6d841b4ba561ec431d29864" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:52.996243 25419 leader_election.cc:306] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:52.996421 25727 tablet_bootstrap.cc:427] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:52.996639 25775 leader_election.cc:243] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:52.996794 25727 ts_tablet_manager.cc:1045] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.140s	user 0.003s	sys 0.001s
I0619 00:11:52.998199 25069 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 98 ms (attempt = 3)
W0619 00:11:52.998474 25069 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:52.998049 25727 raft_consensus.cc:325] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.998703 25727 raft_consensus.cc:351] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:52.996713 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:52.998885 25727 raft_consensus.cc:628] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:52.999517 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6040e369e0fb445fb993d12dd883039b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:52.999305 25727 consensus_queue.cc:227] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:52.999989 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_7846517150f341d0b6fc8df41c79b3a0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.001101 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7846517150f341d0b6fc8df41c79b3a0 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:53.003332 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.004406 25080 catalog_manager.cc:3663] Got report from deleted tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293): Sending delete request for this tablet
I0619 00:11:53.005082 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
I0619 00:11:53.007151 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:53.008965 25336 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.009043 25080 catalog_manager.cc:3663] Got report from deleted tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293): Sending delete request for this tablet
I0619 00:11:53.009707 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c18355df78d344839a56189c7549ab1f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 7277051d52dd4be1a6dfa06488c98293)
W0619 00:11:53.010514 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.011184 25290 consensus_peers.cc:434] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet 70b9c8c4af0945b69b5577e47b61c054. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.011440 25727 ts_tablet_manager.cc:1073] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.001s
I0619 00:11:53.011854 25336 tablet_service.cc:800] Processing DeleteTablet for tablet c18355df78d344839a56189c7549ab1f with delete_type TABLET_DATA_DELETED (replaced by tablet 7277051d52dd4be1a6dfa06488c98293) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.011983 25727 ts_tablet_manager.cc:1028] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:53.012557 25727 tablet_bootstrap.cc:427] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:53.013386 25725 ts_tablet_manager.cc:1073] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.044s	user 0.003s	sys 0.001s
I0619 00:11:53.013816 25725 ts_tablet_manager.cc:1028] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:53.014235 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1cfd335c18c047fba2facc5194807abb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.014912 25725 tablet_bootstrap.cc:427] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:53.016149 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.016319 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 1043 ms (attempt = 7)
W0619 00:11:53.016613 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:53.017829 25741 ts_tablet_manager.cc:1403] T f372b9fb174844eca1662c62039a3c71 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:53.025863 25071 catalog_manager.cc:2891] Scheduling retry of ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC with a delay of 301 ms (attempt = 5)
I0619 00:11:53.026258 25068 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC with a delay of 269 ms (attempt = 5)
W0619 00:11:53.026566 25068 catalog_manager.cc:2910] Async tablet task 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
W0619 00:11:52.985291 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:52.998503 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "397ae10bb6d841b4ba561ec431d29864" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.026971 25622 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.057s (vote granted)
I0619 00:11:53.027220 25622 raft_consensus.cc:2245] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:53.027829 25727 tablet_bootstrap.cc:578] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:52.966365 25623 raft_consensus.cc:2245] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:53.028746 25470 ts_tablet_manager.cc:1173] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.030012 25423 leader_election.cc:379] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.030277 25423 leader_election.cc:270] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:53.030756 25775 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.560s (election complete)
I0619 00:11:53.031025 25775 raft_consensus.cc:2539] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:53.031262 25775 raft_consensus.cc:457] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:53.031270 25423 leader_election.cc:379] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.031625 25423 leader_election.cc:270] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:53.032107 25796 raft_consensus.cc:2732] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.572s (election complete)
I0619 00:11:53.032341 25796 raft_consensus.cc:2539] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:53.032714 25796 raft_consensus.cc:593] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:53.031567 25775 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.781s (starting election)
I0619 00:11:53.033234 25775 raft_consensus.cc:2784] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.033144 25796 consensus_queue.cc:204] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.028091 25071 catalog_manager.cc:2910] Async tablet task ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ee4c4769f75141f9b2ef864ab8a8d9e0 consensus state has no leader
I0619 00:11:53.037677 25775 raft_consensus.cc:479] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.038628 25775 leader_election.cc:243] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:53.040210 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.040623 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "397ae10bb6d841b4ba561ec431d29864" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:53.042162 25419 leader_election.cc:306] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.042593 25775 leader_election.cc:243] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.049388 25727 tablet_bootstrap.cc:427] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:53.049728 25727 ts_tablet_manager.cc:1045] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.037s	user 0.005s	sys 0.000s
I0619 00:11:53.051090 25727 raft_consensus.cc:325] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.051728 25727 raft_consensus.cc:351] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.052767 25336 tablet_service.cc:759] Processing CreateTablet for tablet 29ee1c11105e4d1e9ac74a247cd2953d (table=default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.051913 25727 raft_consensus.cc:628] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:53.053686 25727 consensus_queue.cc:227] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.054574 25082 catalog_manager.cc:3786] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:53.055634 25082 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
I0619 00:11:53.057752 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 99 ms (attempt = 3)
W0619 00:11:53.058440 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:53.059135 25727 ts_tablet_manager.cc:1073] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
W0619 00:11:53.060542 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.062942 25727 ts_tablet_manager.cc:1028] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:53.063380 25727 tablet_bootstrap.cc:427] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:53.063339 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.065882 25727 tablet_bootstrap.cc:578] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:53.067822 25725 tablet_bootstrap.cc:578] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:53.069674 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1cfd335c18c047fba2facc5194807abb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.070299 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6040e369e0fb445fb993d12dd883039b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.070370 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "397ae10bb6d841b4ba561ec431d29864" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:53.070850 25623 raft_consensus.cc:2784] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.071104 25623 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.390s (vote granted)
I0619 00:11:53.071753 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 29ee1c11105e4d1e9ac74a247cd2953d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.073292 25325 ts_tablet_manager.cc:1173] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.074944 25724 tablet_bootstrap.cc:578] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:53.075781 25623 raft_consensus.cc:2245] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:53.076536 25323 ts_tablet_manager.cc:1173] T 1cfd335c18c047fba2facc5194807abb P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.077278 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 198) took 1204ms. Request Metrics: {"mutex_wait_us":46,"tablet-delete.queue_time_us":818858}
I0619 00:11:53.078218 25741 tablet_replica.cc:261] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:53.078241 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) successfully deleted
I0619 00:11:53.078454 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.078672 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.078857 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.079052 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.079268 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.079465 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.079672 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.079851 25741 raft_consensus.cc:2042] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.080102 25741 raft_consensus.cc:2069] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.076942 25423 leader_election.cc:379] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.081060 25423 leader_election.cc:270] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:53.081652 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6040e369e0fb445fb993d12dd883039b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.082469 25796 raft_consensus.cc:2732] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.705s (election complete)
I0619 00:11:53.082731 25796 raft_consensus.cc:2539] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
W0619 00:11:53.084879 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.094738 25775 raft_consensus.cc:457] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.095162 25775 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.530s (starting election)
I0619 00:11:53.096251 25741 ts_tablet_manager.cc:1369] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.095587 25775 raft_consensus.cc:479] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.098923 25069 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 159 ms (attempt = 4)
W0619 00:11:53.099035 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.099241 25069 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:53.099742 25847 raft_consensus.cc:457] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.100005 25847 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.080s (starting election)
I0619 00:11:53.100248 25847 raft_consensus.cc:479] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.101189 25847 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.101605 25847 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.102928 25730 ts_tablet_manager.cc:1382] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.099568 25775 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.103178 25730 log.cc:986] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:53.103222 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:53.103636 25730 ts_tablet_manager.cc:1403] T 7f60ea5c5def47ee846d0186bbc90c18 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:11:53.105043 25419 leader_election.cc:306] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.105037 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 128) took 1523ms. Request Metrics: {"tablet-delete.queue_time_us":1220288}
I0619 00:11:53.105845 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 129) took 1501ms. Request Metrics: {"tablet-delete.queue_time_us":1501171}
I0619 00:11:53.105136 25775 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.106462 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:53.107491 25336 ts_tablet_manager.cc:1173] T 29ee1c11105e4d1e9ac74a247cd2953d P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:53.107815 25419 leader_election.cc:306] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:53.109458 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.110605 25796 raft_consensus.cc:593] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:53.110985 25081 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
I0619 00:11:53.111363 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.111129 25796 consensus_queue.cc:204] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.111855 25623 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.265s (vote granted)
I0619 00:11:53.112083 25623 raft_consensus.cc:2245] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:53.113790 25725 tablet_bootstrap.cc:427] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:53.114148 25725 ts_tablet_manager.cc:1045] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.099s	user 0.005s	sys 0.000s
I0619 00:11:53.114300 25602 ts_tablet_manager.cc:1173] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.116636 25725 raft_consensus.cc:325] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.117297 25725 raft_consensus.cc:351] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.117521 25725 raft_consensus.cc:628] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:53.118342 25899 tablet.cc:1195] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
I0619 00:11:53.118070 25725 consensus_queue.cc:227] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.118906 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 1025 ms (attempt = 7)
W0619 00:11:53.119182 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:53.120172 25736 ts_tablet_manager.cc:1382] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.120419 25736 log.cc:986] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:53.120916 25736 ts_tablet_manager.cc:1403] T cb57d196f0724247b6a7f9a34be8377a P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:53.122361 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 177) took 1239ms. Request Metrics: {"tablet-delete.queue_time_us":997590}
I0619 00:11:53.122891 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 286 ms (attempt = 5)
W0619 00:11:53.123138 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:53.123780 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 176) took 1241ms. Request Metrics: {"tablet-delete.queue_time_us":1237443}
I0619 00:11:53.124367 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 173) took 1242ms. Request Metrics: {"tablet-delete.queue_time_us":1237881}
I0619 00:11:53.124938 25736 tablet_replica.cc:261] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:53.125174 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.125385 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.125586 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.125820 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.126056 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.126277 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.126200 25083 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b] alter complete (version 1)
I0619 00:11:53.126479 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.129983 25931 raft_consensus.cc:457] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.130288 25931 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.204s (starting election)
W0619 00:11:53.121254 25290 consensus_peers.cc:434] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 70b9c8c4af0945b69b5577e47b61c054. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.131376 25468 ts_tablet_manager.cc:1173] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.130738 25931 raft_consensus.cc:479] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.132503 25600 ts_tablet_manager.cc:1173] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.106467 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 130) took 1502ms. Request Metrics: {"tablet-delete.queue_time_us":1501386}
I0619 00:11:53.133662 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 135) took 1278ms. Request Metrics: {"tablet-delete.queue_time_us":1278025}
I0619 00:11:53.134469 25730 tablet_replica.cc:261] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:53.134742 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.134970 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.135184 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(35acf55f35064c1491122a13ff9f5744)
W0619 00:11:53.135324 25419 consensus_peers.cc:434] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 3722b203a516466489c74ef6013eacae. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:53.135408 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.136245 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.136487 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(35acf55f35064c1491122a13ff9f5744)
I0619 00:11:53.136739 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(35acf55f35064c1491122a13ff9f5744)
W0619 00:11:53.137708 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.138976 25423 leader_election.cc:379] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.139232 25423 leader_election.cc:270] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:53.140234 25847 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.100s (election complete)
I0619 00:11:53.126667 25736 raft_consensus.cc:2042] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:53.140424 25083 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:53.140470 25847 raft_consensus.cc:2539] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:53.138447 25724 tablet_bootstrap.cc:427] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:53.140786 25847 raft_consensus.cc:457] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:53.140962 25724 ts_tablet_manager.cc:1045] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.157s	user 0.002s	sys 0.003s
I0619 00:11:53.140995 25847 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.796s (starting election)
I0619 00:11:53.141192 25847 raft_consensus.cc:2784] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.141301 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:53.142439 25923 tablet.cc:1195] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
I0619 00:11:53.139812 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.144088 25623 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.862s (vote granted)
W0619 00:11:53.144903 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.145264 25623 raft_consensus.cc:2245] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:53.145675 25847 raft_consensus.cc:479] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.146668 25847 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.147135 25847 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.147670 25736 consensus_peers.cc:456] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:53.140576 25730 raft_consensus.cc:2042] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.148145 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:53.148633 25736 consensus_peers.cc:456] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:53.148362 25730 raft_consensus.cc:2069] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.149492 25736 raft_consensus.cc:2069] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:53.151290 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times.
I0619 00:11:53.151880 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) successfully deleted
I0619 00:11:53.152513 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet 54373917567a479d93dc8a3fd006d2fe from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:53.152734 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 775f0b6b309c49278066a5e7cf623af2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 775f0b6b309c49278066a5e7cf623af2 already in progress: creating tablet
I0619 00:11:53.152961 25071 catalog_manager.cc:2891] Scheduling retry of 775f0b6b309c49278066a5e7cf623af2 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 36 ms (attempt = 1)
W0619 00:11:53.149756 25419 leader_election.cc:306] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.153959 25081 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_81b1d81c6c2b4710915f7ea29e3ec00b" } new_table_name: "default.table_fd67adcd1cd54957a8ffbe59f5f3e833" alter_external_catalogs: true
I0619 00:11:53.154434 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) successfully deleted
I0619 00:11:53.132292 25931 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:53.155232 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 54373917567a479d93dc8a3fd006d2fe with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 54373917567a479d93dc8a3fd006d2fe already in progress: creating tablet
I0619 00:11:53.155393 25931 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.155443 25071 catalog_manager.cc:2891] Scheduling retry of 54373917567a479d93dc8a3fd006d2fe Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 63 ms (attempt = 1)
I0619 00:11:53.155954 25602 tablet_service.cc:759] Processing CreateTablet for tablet 29ee1c11105e4d1e9ac74a247cd2953d (table=default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.156579 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 29ee1c11105e4d1e9ac74a247cd2953d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.156859 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:53.158514 25419 leader_election.cc:306] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.159809 25725 ts_tablet_manager.cc:1073] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.045s	user 0.004s	sys 0.000s
I0619 00:11:53.160212 25725 ts_tablet_manager.cc:1028] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:53.160770 25725 tablet_bootstrap.cc:427] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:53.163259 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.163180 25724 raft_consensus.cc:325] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.163898 25724 raft_consensus.cc:351] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.164089 25724 raft_consensus.cc:628] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:53.164670 25724 consensus_queue.cc:227] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.167065 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.169857 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 128 ms (attempt = 4)
W0619 00:11:53.170130 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
W0619 00:11:53.170265 25290 consensus_peers.cc:434] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet cb57d196f0724247b6a7f9a34be8377a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.170042 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:53.170760 25623 raft_consensus.cc:2784] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.171000 25623 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.615s (vote granted)
I0619 00:11:53.170599 25730 ts_tablet_manager.cc:1369] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.172853 25081 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 54 ms (attempt = 1)
W0619 00:11:53.173104 25081 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 has no consensus state
I0619 00:11:53.173971 25468 tablet_service.cc:759] Processing CreateTablet for tablet 29ee1c11105e4d1e9ac74a247cd2953d (table=default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.174717 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 29ee1c11105e4d1e9ac74a247cd2953d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.175101 25081 catalog_manager.cc:3663] Got report from deleted tablet 775f0b6b309c49278066a5e7cf623af2 (table default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:53.175799 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:53.175938 25727 tablet_bootstrap.cc:427] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:53.176290 25727 ts_tablet_manager.cc:1045] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.113s	user 0.004s	sys 0.001s
I0619 00:11:53.178032 25727 raft_consensus.cc:325] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.178653 25290 consensus_peers.cc:434] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet cb57d196f0724247b6a7f9a34be8377a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.179464 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.179921 25622 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.035s (vote granted)
I0619 00:11:53.180130 25622 raft_consensus.cc:2245] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:53.180809 25885 raft_consensus.cc:457] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.181041 25885 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.110s (starting election)
I0619 00:11:53.181282 25885 raft_consensus.cc:479] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.181984 25741 ts_tablet_manager.cc:1382] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:53.182216 25741 log.cc:986] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.182248 25885 leader_election.cc:243] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.182795 25741 ts_tablet_manager.cc:1403] T 35acf55f35064c1491122a13ff9f5744 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:53.183060 25885 leader_election.cc:243] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.184197 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a33940e5347489993ab4679e9aeafde" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:53.185627 25553 leader_election.cc:306] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.187768 25623 raft_consensus.cc:2245] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
W0619 00:11:53.188756 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.190904 25068 catalog_manager.cc:2910] Async tablet task 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c57c5ca55f24621b66197a3698504bd consensus state has no leader
I0619 00:11:53.191551 25899 tablet.cc:1195] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
I0619 00:11:53.191751 25354 raft_consensus.cc:2784] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.195798 25602 ts_tablet_manager.cc:1173] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.199012 25725 tablet_bootstrap.cc:578] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:53.200793 25736 ts_tablet_manager.cc:1369] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.201052 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:53.201759 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:53.201917 25081 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_673676f1e1e94d76afe2c9b06d701801" }
W0619 00:11:53.202193 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 53688f0e4a604bc4a07dec9278a8dad6 because the tablet was not found. No further retry: Not found: Tablet not found: 53688f0e4a604bc4a07dec9278a8dad6
I0619 00:11:53.202255 25354 raft_consensus.cc:1081] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W0619 00:11:53.203210 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 54373917567a479d93dc8a3fd006d2fe with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 54373917567a479d93dc8a3fd006d2fe already in progress: creating tablet
I0619 00:11:53.166316 25724 ts_tablet_manager.cc:1073] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.000s
I0619 00:11:53.204375 25775 consensus_queue.cc:963] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:53.206816 25724 ts_tablet_manager.cc:1028] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:53.207507 25724 tablet_bootstrap.cc:427] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:53.208045 25336 tablet_service.cc:759] Processing CreateTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f (table=default.table_559173a19fea4dfea574c9469397df05 [id=2b7639db7e3441539055b4caee80db07]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.209992 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cb3c60fa2d8f4cafa12a5b0bc806433f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.210314 25323 tablet_service.cc:759] Processing CreateTablet for tablet f155a418d0d547f7b775360f1a90ce01 (table=default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.210913 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f155a418d0d547f7b775360f1a90ce01. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.213627 25724 tablet_bootstrap.cc:578] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:53.178647 25727 raft_consensus.cc:351] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:53.215674 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.215694 25727 raft_consensus.cc:628] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:53.216312 25727 consensus_queue.cc:227] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.220582 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:53.221563 25423 leader_election.cc:379] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.221869 25423 leader_election.cc:270] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:53.226176 25727 ts_tablet_manager.cc:1073] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.050s	user 0.004s	sys 0.000s
W0619 00:11:53.227125 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.225376 25323 ts_tablet_manager.cc:1173] T f155a418d0d547f7b775360f1a90ce01 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.234457 25775 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.608s (election complete)
I0619 00:11:53.234727 25775 raft_consensus.cc:2539] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:53.234936 25775 raft_consensus.cc:457] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:53.235132 25775 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.812s (starting election)
I0619 00:11:53.235329 25775 raft_consensus.cc:2784] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.211107 25730 ts_tablet_manager.cc:1382] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:53.235848 25730 log.cc:986] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.236523 25730 ts_tablet_manager.cc:1403] T 35acf55f35064c1491122a13ff9f5744 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:53.203162 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_caad097bfc684f938c5e58f0eaa18870" } new_table_name: "default.table_811056346d9b4f5e8fb98ccc54d55045" alter_external_catalogs: true
I0619 00:11:53.238052 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 136) took 1230ms. Request Metrics: {"tablet-delete.queue_time_us":1122565}
W0619 00:11:53.238350 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.204056 25083 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
I0619 00:11:53.228989 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 6040e369e0fb445fb993d12dd883039b
I0619 00:11:53.231775 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 2094 ms (attempt = 8)
W0619 00:11:53.239001 25067 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:53.238848 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 56 ms (attempt = 2)
I0619 00:11:53.239264 25083 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 17 ms (attempt = 1)
W0619 00:11:53.239540 25083 catalog_manager.cc:3945] Failed to send 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
W0619 00:11:53.239362 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:53.240691 25325 tablet_service.cc:759] Processing CreateTablet for tablet 93930ffe285d48d080a8ca4793e871d9 (table=default.table_3f22e2fb4a9e47aca16e414746eecd63 [id=acb8add3e0fd49fab202731910e3d20e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.238834 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.241269 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 93930ffe285d48d080a8ca4793e871d9. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.240945 25775 raft_consensus.cc:479] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.241369 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.241760 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.241930 25775 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.242311 25775 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.242184 25323 tablet_service.cc:759] Processing CreateTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 (table=default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.242866 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a33940e5347489993ab4679e9aeafde" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.243194 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.243667 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9a221a1d9814ef99a0b4ccaa04d1017. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.244173 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:53.245615 25081 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
W0619 00:11:53.245784 25419 leader_election.cc:306] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.246191 25923 tablet.cc:1195] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
I0619 00:11:53.246510 25081 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 35 ms (attempt = 1)
W0619 00:11:53.246788 25081 catalog_manager.cc:3945] Failed to send 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
W0619 00:11:53.248292 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.248484 25730 tablet_replica.cc:261] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:53.248790 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:53.250363 25727 ts_tablet_manager.cc:1028] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:53.251152 25725 tablet_bootstrap.cc:427] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:53.251516 25725 ts_tablet_manager.cc:1045] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.091s	user 0.005s	sys 0.000s
I0619 00:11:53.253664 25071 catalog_manager.cc:2891] Scheduling retry of 54373917567a479d93dc8a3fd006d2fe Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 32 ms (attempt = 1)
W0619 00:11:53.253913 25071 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 50715.5us
I0619 00:11:53.224923 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 199) took 1254ms. Request Metrics: {"mutex_wait_us":1499,"tablet-delete.queue_time_us":1087540}
I0619 00:11:53.256881 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_5f1d24330f9f41969881d224929b3ee2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.257788 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5f1d24330f9f41969881d224929b3ee2 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.257768 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) successfully deleted
I0619 00:11:53.257990 25727 tablet_bootstrap.cc:427] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:53.258383 25081 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_7937f0a6914a40b8baff07c0983cb44c" }
I0619 00:11:53.256875 25725 raft_consensus.cc:325] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.259224 25725 raft_consensus.cc:351] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.259405 25423 leader_election.cc:379] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.260403 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 34 ms (attempt = 2)
W0619 00:11:53.260686 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:53.261049 25069 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 288 ms (attempt = 5)
W0619 00:11:53.261278 25069 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:53.257824 25741 tablet_replica.cc:261] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:53.260740 25423 leader_election.cc:270] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:53.261749 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:53.261962 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:53.261987 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:53.254932 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.262161 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:53.255350 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.254119 25623 raft_consensus.cc:1081] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:53.262642 25885 raft_consensus.cc:457] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.259436 25725 raft_consensus.cc:628] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:53.262894 25885 raft_consensus.cc:2732] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.623s (starting election)
I0619 00:11:53.263135 25885 raft_consensus.cc:479] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.263293 25725 consensus_queue.cc:227] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.264106 25885 leader_election.cc:243] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.264494 25885 leader_election.cc:243] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.262354 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:53.265238 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:53.265678 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4a88e394b199431e8b794a1b1aec847f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:53.266710 25080 catalog_manager.cc:3663] Got report from deleted tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
W0619 00:11:53.267102 25553 leader_election.cc:306] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.267335 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:53.268108 25796 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.085s (election complete)
I0619 00:11:53.268352 25796 raft_consensus.cc:2539] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:53.268565 25796 raft_consensus.cc:457] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:53.268836 25796 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.922s (starting election)
I0619 00:11:53.269038 25796 raft_consensus.cc:2784] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.269886 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:53.270108 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:53.270335 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:53.270536 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:53.272668 25336 ts_tablet_manager.cc:1173] T cb3c60fa2d8f4cafa12a5b0bc806433f P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:53.273519 25068 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:53.273707 25080 catalog_manager.cc:3663] Got report from deleted tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:53.273749 25796 raft_consensus.cc:479] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.274334 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:53.274641 25796 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.275045 25796 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.276973 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:53.278892 25725 ts_tablet_manager.cc:1073] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.000s
I0619 00:11:53.279265 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:53.279660 25730 raft_consensus.cc:2042] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.285473 25067 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 57 ms (attempt = 2)
W0619 00:11:53.285792 25067 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:53.286981 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:53.287305 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.287720 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.288666 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.289376 25730 raft_consensus.cc:2069] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.290102 25725 ts_tablet_manager.cc:1028] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:53.290181 25724 tablet_bootstrap.cc:427] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
W0619 00:11:53.290398 25558 leader_election.cc:306] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.290812 25558 leader_election.cc:270] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.291268 25885 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.603s (election complete)
I0619 00:11:53.291501 25885 raft_consensus.cc:2487] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.292068 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:53.292590 25623 raft_consensus.cc:2784] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.292598 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:53.292739 25724 ts_tablet_manager.cc:1045] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.085s	user 0.004s	sys 0.001s
I0619 00:11:53.293244 25623 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.365s (vote granted)
I0619 00:11:53.293273 25622 raft_consensus.cc:2784] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.293601 25622 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.581s (vote granted)
I0619 00:11:53.294121 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) successfully deleted
I0619 00:11:53.295965 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 96 ms (attempt = 3)
W0619 00:11:53.296600 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:53.297550 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 97 ms (attempt = 3)
I0619 00:11:53.297722 25602 tablet_service.cc:759] Processing CreateTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f (table=default.table_559173a19fea4dfea574c9469397df05 [id=2b7639db7e3441539055b4caee80db07]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:53.297873 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:53.297891 25622 raft_consensus.cc:2245] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:53.298347 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cb3c60fa2d8f4cafa12a5b0bc806433f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.299538 25600 tablet_service.cc:759] Processing CreateTablet for tablet f155a418d0d547f7b775360f1a90ce01 (table=default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.300118 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f155a418d0d547f7b775360f1a90ce01. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.307945 25068 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC with a delay of 514 ms (attempt = 6)
W0619 00:11:53.308221 25068 catalog_manager.cc:2910] Async tablet task 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
I0619 00:11:53.309641 25600 ts_tablet_manager.cc:1173] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.311553 25601 tablet_service.cc:759] Processing CreateTablet for tablet 93930ffe285d48d080a8ca4793e871d9 (table=default.table_3f22e2fb4a9e47aca16e414746eecd63 [id=acb8add3e0fd49fab202731910e3d20e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.312162 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 93930ffe285d48d080a8ca4793e871d9. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.312398 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:53.312492 25081 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
I0619 00:11:53.312858 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:53.291522 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.292068 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4a88e394b199431e8b794a1b1aec847f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.313966 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_a822c7bace2a4cfd9c1c7f0a9c992551"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:53.314673 25725 tablet_bootstrap.cc:427] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:53.314779 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:53.314887 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a822c7bace2a4cfd9c1c7f0a9c992551 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.317034 25725 tablet_bootstrap.cc:578] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:53.304550 25423 leader_election.cc:379] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.317296 25423 leader_election.cc:270] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:53.300009 25899 tablet.cc:1200] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:53.318747 25602 ts_tablet_manager.cc:1173] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.319630 25931 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.767s (election complete)
I0619 00:11:53.319779 25600 tablet_service.cc:759] Processing CreateTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 (table=default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.319905 25931 raft_consensus.cc:2539] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:53.320204 25931 raft_consensus.cc:593] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:53.320344 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9a221a1d9814ef99a0b4ccaa04d1017. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.319947 25730 ts_tablet_manager.cc:1369] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.320628 25931 consensus_queue.cc:204] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.265637 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:53.277930 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.323532 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(3b80a302aacb418091aee54ab4be7249)
I0619 00:11:53.284755 25727 tablet_bootstrap.cc:578] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:53.303887 25468 ts_tablet_manager.cc:1173] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.323781 25741 raft_consensus.cc:2042] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.324055 25741 raft_consensus.cc:2069] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.293069 25891 raft_consensus.cc:457] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.325026 25891 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.043s (starting election)
I0619 00:11:53.288619 25331 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.325295 25741 ts_tablet_manager.cc:1369] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.325286 25891 raft_consensus.cc:479] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.325819 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.326328 25891 leader_election.cc:243] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:53.326437 25558 leader_election.cc:306] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.326735 25558 leader_election.cc:270] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.326756 25891 leader_election.cc:243] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.326691 25468 tablet_service.cc:759] Processing CreateTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f (table=default.table_559173a19fea4dfea574c9469397df05 [id=2b7639db7e3441539055b4caee80db07]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.327268 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cb3c60fa2d8f4cafa12a5b0bc806433f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.327370 25885 raft_consensus.cc:2732] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.134s (election complete)
I0619 00:11:53.327606 25885 raft_consensus.cc:2487] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.329684 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a85577b0beba4276b963e5c360544bc7" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.330137 25489 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.476s (vote granted)
I0619 00:11:53.330364 25489 raft_consensus.cc:2245] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:53.331512 25290 leader_election.cc:379] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.331758 25290 leader_election.cc:270] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:53.332185 25891 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.051s (election complete)
W0619 00:11:53.332352 25290 leader_election.cc:306] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:53.332906 25914 raft_consensus.cc:457] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.332993 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 283 ms (attempt = 5)
I0619 00:11:53.333133 25914 raft_consensus.cc:2732] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.866s (starting election)
W0619 00:11:53.333274 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:53.333374 25914 raft_consensus.cc:479] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.334661 25914 leader_election.cc:243] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.335000 25914 leader_election.cc:243] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.332420 25891 raft_consensus.cc:2539] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:53.335537 25891 raft_consensus.cc:457] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:53.335767 25891 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.932s (starting election)
I0619 00:11:53.335801 25623 raft_consensus.cc:2245] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:53.335963 25891 raft_consensus.cc:2784] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.307543 25736 ts_tablet_manager.cc:1382] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W0619 00:11:53.340898 25290 leader_election.cc:306] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: c18355df78d344839a56189c7549ab1f
I0619 00:11:53.341823 25325 ts_tablet_manager.cc:1173] T 93930ffe285d48d080a8ca4793e871d9 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.342200 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:53.344208 25067 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 87 ms (attempt = 3)
W0619 00:11:53.344466 25067 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:53.345530 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 1073 ms (attempt = 7)
W0619 00:11:53.345818 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:53.345940 25080 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
W0619 00:11:53.346884 25290 leader_election.cc:306] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: c18355df78d344839a56189c7549ab1f
I0619 00:11:53.347164 25290 leader_election.cc:270] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.295626 25724 raft_consensus.cc:325] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.351336 25323 ts_tablet_manager.cc:1173] T e9a221a1d9814ef99a0b4ccaa04d1017 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.340167 25891 raft_consensus.cc:479] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.352717 25891 leader_election.cc:243] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.353051 25724 raft_consensus.cc:351] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.354955 25724 raft_consensus.cc:628] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:53.353533 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.356827 25891 leader_election.cc:243] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.357257 25891 raft_consensus.cc:2732] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.751s (election complete)
I0619 00:11:53.357388 25724 consensus_queue.cc:227] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.359222 25736 log.cc:986] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.342860 25467 tablet_service.cc:759] Processing CreateTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 (table=default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:53.345371 25419 leader_election.cc:306] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:53.351404 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.352267 25464 tablet_service.cc:759] Processing CreateTablet for tablet 93930ffe285d48d080a8ca4793e871d9 (table=default.table_3f22e2fb4a9e47aca16e414746eecd63 [id=acb8add3e0fd49fab202731910e3d20e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.359823 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9a221a1d9814ef99a0b4ccaa04d1017. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.359797 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a85577b0beba4276b963e5c360544bc7" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:53.360632 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 93930ffe285d48d080a8ca4793e871d9. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.364317 25736 ts_tablet_manager.cc:1403] T 35acf55f35064c1491122a13ff9f5744 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
W0619 00:11:53.365025 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.334136 25470 tablet_service.cc:759] Processing CreateTablet for tablet f155a418d0d547f7b775360f1a90ce01 (table=default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.365528 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f155a418d0d547f7b775360f1a90ce01. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.366762 25489 raft_consensus.cc:2784] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.367067 25489 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.573s (vote granted)
I0619 00:11:53.368969 25087 catalog_manager.cc:3663] Got report from deleted tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:53.369720 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:53.370473 25464 ts_tablet_manager.cc:1173] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:53.371803 25290 leader_election.cc:306] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:53.373744 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:53.374447 25423 leader_election.cc:379] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.374716 25423 leader_election.cc:270] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
W0619 00:11:53.377534 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.375886 25423 leader_election.cc:379] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.377890 25423 leader_election.cc:270] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:53.352984 25775 consensus_queue.cc:963] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:53.378329 25847 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.010s (election complete)
I0619 00:11:53.378571 25847 raft_consensus.cc:2539] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:53.375162 25796 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.169s (election complete)
I0619 00:11:53.379154 25796 raft_consensus.cc:2539] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:53.379443 25796 raft_consensus.cc:593] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:53.386603 25470 ts_tablet_manager.cc:1173] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.388921 25601 ts_tablet_manager.cc:1173] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.389035 25942 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 4
I0619 00:11:53.357491 25891 raft_consensus.cc:2487] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.389946 25891 raft_consensus.cc:457] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.390175 25891 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.774s (starting election)
I0619 00:11:53.390425 25891 raft_consensus.cc:479] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.391070 25741 ts_tablet_manager.cc:1382] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.391305 25741 log.cc:986] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/3b80a302aacb418091aee54ab4be7249
I0619 00:11:53.391419 25891 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.391903 25891 leader_election.cc:243] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.392961 25087 catalog_manager.cc:3663] Got report from deleted tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:53.393668 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:53.393985 25727 tablet_bootstrap.cc:427] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:53.394287 25727 ts_tablet_manager.cc:1045] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.143s	user 0.005s	sys 0.000s
I0619 00:11:53.394384 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 168 ms (attempt = 4)
W0619 00:11:53.394675 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:53.395376 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet a52cbcbe35374d518bca1e0fc24d88d4 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:53.395730 25727 raft_consensus.cc:325] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.395382 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.396309 25727 raft_consensus.cc:351] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.396754 25623 raft_consensus.cc:2206] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:53.396740 25727 raft_consensus.cc:628] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:53.376760 25724 ts_tablet_manager.cc:1073] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.083s	user 0.004s	sys 0.000s
I0619 00:11:53.397987 25724 ts_tablet_manager.cc:1028] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:53.398253 25290 leader_election.cc:393] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:53.398459 25727 consensus_queue.cc:227] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.400800 25727 ts_tablet_manager.cc:1073] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.002s
W0619 00:11:53.401120 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.403029 25943 tablet.cc:1200] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:53.401098 25727 ts_tablet_manager.cc:1028] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:53.408242 25727 tablet_bootstrap.cc:427] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:53.409096 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.410370 25069 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 547 ms (attempt = 6)
W0619 00:11:53.410660 25069 catalog_manager.cc:2910] Async tablet task 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 075076aaa16e4ea8a596a7a8c52f103e consensus state has no leader
I0619 00:11:53.409505 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 129 ms (attempt = 4)
W0619 00:11:53.411170 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:53.415060 25083 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4] alter complete (version 1)
I0619 00:11:53.416213 25724 tablet_bootstrap.cc:427] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:53.418347 25724 tablet_bootstrap.cc:578] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:53.419909 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_fa53e172a2a34136a5a6efb8b5c39368"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.420789 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fa53e172a2a34136a5a6efb8b5c39368 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.379891 25796 consensus_queue.cc:204] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.428095 25467 ts_tablet_manager.cc:1173] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.428187 25847 raft_consensus.cc:593] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:53.429203 25847 consensus_queue.cc:204] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.427839 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 181) took 1418ms. Request Metrics: {"tablet-delete.queue_time_us":1114672}
I0619 00:11:53.432922 25067 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 138 ms (attempt = 4)
W0619 00:11:53.433207 25067 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
I0619 00:11:53.438868 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_638430eb41c8443fb3e0f1ead3b26dcb"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.439877 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_638430eb41c8443fb3e0f1ead3b26dcb in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.431869 25488 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3722b203a516466489c74ef6013eacae" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:53.445834 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 67034.4us
I0619 00:11:53.450425 25082 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 3722b203a516466489c74ef6013eacae (table default.table_fb89f806be98424aa43bd01b3f816a80 [id=a17737f9d2924cc09f372527fb91188b]). Expected version 1 got 0
I0619 00:11:53.442572 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 182) took 1432ms. Request Metrics: {"tablet-delete.queue_time_us":1421705}
I0619 00:11:53.450803 25082 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:53.452076 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:53.452217 25068 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:53.452323 25488 raft_consensus.cc:2206] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
W0619 00:11:53.452699 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.453579 25899 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 4
I0619 00:11:53.455624 25727 tablet_bootstrap.cc:578] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:53.409922 25885 raft_consensus.cc:457] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.457270 25885 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.124s (starting election)
I0619 00:11:53.457537 25885 raft_consensus.cc:479] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.441676 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.458508 25885 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.458937 25885 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.401417 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.405781 25601 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:53.459800 25290 consensus_peers.cc:434] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 03522c48c50943b38cae5aa88562e583. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.459884 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:53.460309 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.460964 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:53.461525 25923 tablet.cc:1200] T fb8e94875a8347a0a473c95a77c38795 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 3 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 4
I0619 00:11:53.447232 25487 raft_consensus.cc:1081] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:11:53.462404 25553 leader_election.cc:306] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.462841 25470 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:53.463098 25082 master_service.cc:523] ReplaceTablet: received request to replace tablet a4c653a255984d3bac88df63362b9334 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:53.463769 25081 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_43102886b675447088b4319ef7a3b947" } new_table_name: "default.table_1074470b12de48baa6a1264ad788d237" alter_external_catalogs: true
I0619 00:11:53.465116 25489 raft_consensus.cc:2245] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:53.466250 25741 ts_tablet_manager.cc:1403] T 3b80a302aacb418091aee54ab4be7249 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:53.467442 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.468678 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 202) took 1421ms. Request Metrics: {"tablet-delete.queue_time_us":1210034}
I0619 00:11:53.467866 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 185) took 1438ms. Request Metrics: {"tablet-delete.queue_time_us":1437723}
I0619 00:11:53.469269 25736 tablet_replica.cc:261] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:53.469506 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:53.469324 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 203) took 1420ms. Request Metrics: {"tablet-delete.queue_time_us":1420327}
I0619 00:11:53.469733 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:53.469926 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:53.469946 25741 tablet_replica.cc:261] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:53.470124 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:53.470166 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.470343 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:53.470371 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.470544 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:53.470752 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:53.460949 25290 leader_election.cc:393] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:53.470932 25736 raft_consensus.cc:2042] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.471043 25290 leader_election.cc:270] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.471037 25468 ts_tablet_manager.cc:1173] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.471191 25736 raft_consensus.cc:2069] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.471609 25891 raft_consensus.cc:2732] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.032s (election complete)
I0619 00:11:53.471894 25891 raft_consensus.cc:2784] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.472414 25899 tablet.cc:1195] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
I0619 00:11:53.474544 25468 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:53.476279 25891 raft_consensus.cc:2487] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:53.478735 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.487913 25600 ts_tablet_manager.cc:1173] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.488574 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 35acf55f35064c1491122a13ff9f5744 (table default.table_e2e6deb5e5764586802390a904ba068f [id=76428d4b5c9a469998b9b5fe01c1d104]) successfully deleted
I0619 00:11:53.488066 25730 ts_tablet_manager.cc:1382] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.483317 25323 tablet_service.cc:759] Processing CreateTablet for tablet 4feb648bb30247b49a4abd49da95ea7d (table=default.table_5f1d24330f9f41969881d224929b3ee2 [id=b9b59eba60624c4bbde989c6c7872e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.491572 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4feb648bb30247b49a4abd49da95ea7d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.489095 25730 log.cc:986] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/397ae10bb6d841b4ba561ec431d29864
W0619 00:11:53.493113 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.494787 25736 ts_tablet_manager.cc:1369] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.497880 25725 tablet_bootstrap.cc:427] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:53.498522 25725 ts_tablet_manager.cc:1045] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.184s	user 0.005s	sys 0.005s
I0619 00:11:53.470569 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.499194 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.499337 25730 ts_tablet_manager.cc:1403] T 397ae10bb6d841b4ba561ec431d29864 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:53.499549 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.499876 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.499985 25290 leader_election.cc:379] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.501493 25468 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:53.502143 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.503298 25290 leader_election.cc:270] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:53.500389 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.504600 25741 raft_consensus.cc:2042] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.505028 25741 raft_consensus.cc:2069] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.506242 25083 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_638430eb41c8443fb3e0f1ead3b26dcb" }
I0619 00:11:53.508740 25727 tablet_bootstrap.cc:427] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:53.509096 25727 ts_tablet_manager.cc:1045] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.101s	user 0.005s	sys 0.003s
I0619 00:11:53.509469 25725 raft_consensus.cc:325] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.510113 25725 raft_consensus.cc:351] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.510306 25725 raft_consensus.cc:628] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:53.510704 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:53.514179 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.514561 25725 consensus_queue.cc:227] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.517890 25080 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_811056346d9b4f5e8fb98ccc54d55045 [id=41f48f0e1f87464eb43c5bf2a36f70ef] alter complete (version 4)
I0619 00:11:53.518715 25081 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC with a delay of 62 ms (attempt = 1)
W0619 00:11:53.518998 25081 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc7686daae274e0dba87703fe6924805 consensus state has no leader
I0619 00:11:53.521459 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.523023 25600 tablet_service.cc:759] Processing CreateTablet for tablet 4feb648bb30247b49a4abd49da95ea7d (table=default.table_5f1d24330f9f41969881d224929b3ee2 [id=b9b59eba60624c4bbde989c6c7872e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.525918 25940 raft_consensus.cc:2732] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.191s (election complete)
I0619 00:11:53.526218 25940 raft_consensus.cc:2539] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:53.526533 25940 raft_consensus.cc:593] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:53.526815 25914 consensus_queue.cc:963] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:53.527256 25083 catalog_manager.cc:4020] Not sending DeleteTablet requests; no consensus state for tablet ec4c6e2b7eb945ebade21bf6ae119c3c
I0619 00:11:53.527056 25940 consensus_queue.cc:204] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.528288 25727 raft_consensus.cc:325] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.529002 25727 raft_consensus.cc:351] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.529377 25727 raft_consensus.cc:628] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:53.534183 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4feb648bb30247b49a4abd49da95ea7d. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:53.506340 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.536865 25741 ts_tablet_manager.cc:1369] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:53.538007 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a because the tablet was not found. No further retry: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
W0619 00:11:53.538445 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times.
I0619 00:11:53.538221 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ddf8a60eb43545bda4bfaea6216dba67"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.539180 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ddf8a60eb43545bda4bfaea6216dba67 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.500561 25323 ts_tablet_manager.cc:1173] T 4feb648bb30247b49a4abd49da95ea7d P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.540539 25354 raft_consensus.cc:2784] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.530050 25727 consensus_queue.cc:227] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.541100 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.542332 25923 tablet.cc:1200] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:53.523134 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 139) took 1514ms. Request Metrics: {"tablet-delete.queue_time_us":1236092}
I0619 00:11:53.544126 25730 tablet_replica.cc:261] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:53.544945 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 303 ms (attempt = 5)
W0619 00:11:53.545204 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:53.547794 25354 raft_consensus.cc:1081] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:53.549657 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.550357 25069 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 541 ms (attempt = 6)
W0619 00:11:53.550642 25069 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_970d5da055774f58881ea62e91e2f1f4 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:53.550206 25796 consensus_queue.cc:963] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I0619 00:11:53.552455 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.554231 25724 tablet_bootstrap.cc:427] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:53.557176 25724 ts_tablet_manager.cc:1045] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.141s	user 0.006s	sys 0.001s
I0619 00:11:53.558481 25724 raft_consensus.cc:325] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.559185 25724 raft_consensus.cc:351] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.559393 25724 raft_consensus.cc:628] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:53.559952 25724 consensus_queue.cc:227] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.562073 25727 ts_tablet_manager.cc:1073] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.053s	user 0.004s	sys 0.000s
I0619 00:11:53.562868 25082 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]). Expected version 1 got 0
I0619 00:11:53.563405 25069 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 258 ms (attempt = 5)
W0619 00:11:53.563671 25069 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
W0619 00:11:53.563766 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.564468 25727 ts_tablet_manager.cc:1028] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:53.565307 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.565675 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.568186 25891 raft_consensus.cc:457] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.559067 25725 ts_tablet_manager.cc:1073] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.060s	user 0.004s	sys 0.000s
I0619 00:11:53.568706 25725 ts_tablet_manager.cc:1028] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:53.569223 25891 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.860s (starting election)
W0619 00:11:53.569314 25558 leader_election.cc:306] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.569663 25558 leader_election.cc:270] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.570482 25725 tablet_bootstrap.cc:427] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:53.570636 25885 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.717s (election complete)
I0619 00:11:53.570883 25885 raft_consensus.cc:2487] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.570449 25942 tablet.cc:1195] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
I0619 00:11:53.571640 25067 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 284 ms (attempt = 5)
W0619 00:11:53.571961 25067 catalog_manager.cc:2910] Async tablet task 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 30a9d9fda8d34caeb8242c372439321b consensus state has no leader
W0619 00:11:53.572351 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.574306 25468 tablet_service.cc:759] Processing CreateTablet for tablet 4feb648bb30247b49a4abd49da95ea7d (table=default.table_5f1d24330f9f41969881d224929b3ee2 [id=b9b59eba60624c4bbde989c6c7872e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.574920 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4feb648bb30247b49a4abd49da95ea7d. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:53.574985 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.576393 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.572736 25891 raft_consensus.cc:479] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.565042 25727 tablet_bootstrap.cc:427] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:53.578810 25724 ts_tablet_manager.cc:1073] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I0619 00:11:53.579145 25724 ts_tablet_manager.cc:1028] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:53.579277 25891 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.579484 25724 tablet_bootstrap.cc:427] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:53.580726 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) successfully deleted
W0619 00:11:53.579880 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.580452 25290 consensus_peers.cc:434] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet 03522c48c50943b38cae5aa88562e583. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.581542 25891 leader_election.cc:243] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.581701 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.582033 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.582252 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.582458 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.582626 25730 raft_consensus.cc:2042] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.582903 25730 raft_consensus.cc:2069] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.585088 25730 ts_tablet_manager.cc:1369] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.587663 25947 raft_consensus.cc:457] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.587929 25947 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.992s (starting election)
I0619 00:11:53.588181 25947 raft_consensus.cc:479] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.589700 25736 ts_tablet_manager.cc:1382] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.589951 25736 log.cc:986] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/03522c48c50943b38cae5aa88562e583
I0619 00:11:53.590414 25736 ts_tablet_manager.cc:1403] T 03522c48c50943b38cae5aa88562e583 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
W0619 00:11:53.593366 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.593716 25468 ts_tablet_manager.cc:1173] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.596150 25724 tablet_bootstrap.cc:578] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:53.596513 25082 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:53.597582 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:53.597571 25947 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.598060 25947 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.598685 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.599275 25487 raft_consensus.cc:2206] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:53.599928 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:53.600555 25725 tablet_bootstrap.cc:578] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:53.601322 25419 consensus_peers.cc:434] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet e3708e05f6054306b5bf7f1df2140e3d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:11:53.602319 25419 leader_election.cc:306] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.591853 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 187) took 1496ms. Request Metrics: {"tablet-delete.queue_time_us":1346374}
I0619 00:11:53.603353 25736 tablet_replica.cc:261] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:53.603593 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.603828 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.604022 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.604204 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.604420 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.604642 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.604837 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(e3708e05f6054306b5bf7f1df2140e3d)
I0619 00:11:53.605242 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.605741 25622 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.659s (vote granted)
I0619 00:11:53.605947 25622 raft_consensus.cc:2245] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:53.607277 25923 tablet.cc:1195] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
W0619 00:11:53.610950 25748 thread.cc:554] rpc reactor (reactor) Time spent creating pthread: real 1.270s	user 0.001s	sys 0.002s
W0619 00:11:53.611162 25748 thread.cc:521] rpc reactor (reactor) Time spent starting thread: real 1.270s	user 0.001s	sys 0.002s
W0619 00:11:53.612365 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.617153 25600 ts_tablet_manager.cc:1173] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.622185 25741 ts_tablet_manager.cc:1382] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I0619 00:11:53.626969 25741 log.cc:986] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/e3708e05f6054306b5bf7f1df2140e3d
I0619 00:11:53.627621 25741 ts_tablet_manager.cc:1403] T e3708e05f6054306b5bf7f1df2140e3d P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:53.629503 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC with a delay of 81 ms (attempt = 2)
W0619 00:11:53.633635 25071 catalog_manager.cc:2910] Async tablet task fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fc7686daae274e0dba87703fe6924805 consensus state has no leader
W0619 00:11:53.617743 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.634202 25736 raft_consensus.cc:2042] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:53.634912 25736 consensus_peers.cc:456] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:53.635499 25736 consensus_peers.cc:456] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:53.636502 25736 raft_consensus.cc:2069] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.636504 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 561 ms (attempt = 6)
W0619 00:11:53.636831 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:53.636832 25290 leader_election.cc:393] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:53.646348 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) successfully deleted
I0619 00:11:53.630180 25724 tablet_bootstrap.cc:427] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:53.636143 25885 raft_consensus.cc:457] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.649415 25885 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.018s (starting election)
I0619 00:11:53.645007 25601 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.648828 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_0075a29451044e27990e84f7ae19482d"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:53.643199 25796 raft_consensus.cc:457] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.649533 25724 ts_tablet_manager.cc:1045] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.070s	user 0.004s	sys 0.001s
I0619 00:11:53.649883 25796 raft_consensus.cc:2732] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.132s (starting election)
W0619 00:11:53.650152 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0075a29451044e27990e84f7ae19482d in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.649729 25885 raft_consensus.cc:479] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.650172 25796 raft_consensus.cc:479] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.650787 25885 leader_election.cc:243] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.651149 25796 leader_election.cc:243] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.651191 25885 leader_election.cc:243] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.651558 25796 leader_election.cc:243] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.652156 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_ddf8a60eb43545bda4bfaea6216dba67" } new_table_name: "default.table_bd5ecdaeb9784fe696613bd457e08878" alter_external_catalogs: true
I0619 00:11:53.653090 25730 ts_tablet_manager.cc:1382] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I0619 00:11:53.653517 25730 log.cc:986] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/e3708e05f6054306b5bf7f1df2140e3d
I0619 00:11:53.653424 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a52cbcbe35374d518bca1e0fc24d88d4" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:53.654146 25730 ts_tablet_manager.cc:1403] T e3708e05f6054306b5bf7f1df2140e3d P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:53.654330 25736 ts_tablet_manager.cc:1369] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.654672 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_fa53e172a2a34136a5a6efb8b5c39368" }
W0619 00:11:53.654916 25419 leader_election.cc:306] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.655805 25725 tablet_bootstrap.cc:427] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:53.629204 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 204) took 1579ms. Request Metrics: {"tablet-delete.queue_time_us":1419791}
I0619 00:11:53.655318 25724 raft_consensus.cc:325] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.657037 25724 raft_consensus.cc:351] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.657244 25724 raft_consensus.cc:628] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:53.657297 25725 ts_tablet_manager.cc:1045] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.087s	user 0.007s	sys 0.002s
I0619 00:11:53.657804 25724 consensus_queue.cc:227] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.659163 25725 raft_consensus.cc:325] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.659804 25725 raft_consensus.cc:351] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:53.660758 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.661048 25725 raft_consensus.cc:628] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:53.662461 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.663583 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) successfully deleted
I0619 00:11:53.656600 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 206) took 1530ms. Request Metrics: {"tablet-delete.queue_time_us":1529104}
I0619 00:11:53.664670 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775f0b6b309c49278066a5e7cf623af2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:53.666005 25553 leader_election.cc:306] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.661963 25725 consensus_queue.cc:227] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.621462 25423 leader_election.cc:379] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.669142 25423 leader_election.cc:270] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:53.669656 25796 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.637s (election complete)
I0619 00:11:53.669973 25796 raft_consensus.cc:2539] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:53.670176 25796 raft_consensus.cc:457] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:53.670377 25796 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.750s (starting election)
I0619 00:11:53.670646 25725 ts_tablet_manager.cc:1073] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I0619 00:11:53.670611 25796 raft_consensus.cc:2784] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.671041 25725 ts_tablet_manager.cc:1028] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:53.671389 25725 tablet_bootstrap.cc:427] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:53.667593 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.674413 25725 tablet_bootstrap.cc:578] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:53.675894 25067 catalog_manager.cc:2910] Async tablet task d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet d62715e230154e2abe126ef97feec7ce consensus state has no leader
I0619 00:11:53.676759 25081 catalog_manager.cc:3786] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:53.678232 25081 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:53.678725 25081 catalog_manager.cc:3786] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:53.682989 25323 tablet_service.cc:759] Processing CreateTablet for tablet 8e3a356933e749b9a4ade476ec30f475 (table=default.table_a822c7bace2a4cfd9c1c7f0a9c992551 [id=0d90b232db8e48ed9bfab0c59c6fbdba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.683559 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8e3a356933e749b9a4ade476ec30f475. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.685559 25083 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 44 ms (attempt = 1)
I0619 00:11:53.685061 25081 catalog_manager.cc:3786] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
W0619 00:11:53.685853 25083 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c has no consensus state
I0619 00:11:53.688066 25724 ts_tablet_manager.cc:1073] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.000s
W0619 00:11:53.688875 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 67451.4us
I0619 00:11:53.690665 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_252d88d7d98b495e80affd5dc6eeb739"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:53.691365 25736 ts_tablet_manager.cc:1382] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W0619 00:11:53.691632 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_252d88d7d98b495e80affd5dc6eeb739 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.692376 25325 tablet_service.cc:759] Processing CreateTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 (table=default.table_fa53e172a2a34136a5a6efb8b5c39368 [id=779e902dec1a4701a176d545a35e9978]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.692751 25736 log.cc:986] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/e3708e05f6054306b5bf7f1df2140e3d
I0619 00:11:53.692984 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f75b8a4f1fd845fbab6f83f59b7cfc58. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.693395 25736 ts_tablet_manager.cc:1403] T e3708e05f6054306b5bf7f1df2140e3d P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:53.695449 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_f749a3c527f5449f8d30ae159bf2a2cb"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.696331 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f749a3c527f5449f8d30ae159bf2a2cb in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.697109 25081 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
I0619 00:11:53.699962 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 141) took 1637ms. Request Metrics: {"mutex_wait_us":5015,"tablet-delete.queue_time_us":1465765}
I0619 00:11:53.700698 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 140) took 1638ms. Request Metrics: {"tablet-delete.queue_time_us":1622200}
I0619 00:11:53.664206 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 207) took 1527ms. Request Metrics: {"tablet-delete.queue_time_us":1526447}
W0619 00:11:53.701550 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.706993 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 143) took 1537ms. Request Metrics: {"tablet-delete.queue_time_us":1536789}
I0619 00:11:53.707559 25730 tablet_replica.cc:261] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:53.707826 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:53.708037 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:53.708241 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:53.708446 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:53.708699 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:53.708899 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:53.709079 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:53.701494 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 208) took 1559ms. Request Metrics: {"tablet-delete.queue_time_us":1558717}
I0619 00:11:53.710141 25727 tablet_bootstrap.cc:578] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
W0619 00:11:53.710860 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet edfdedbd4eca4b62be24b1649f820d0e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet edfdedbd4eca4b62be24b1649f820d0e already in progress: creating tablet
I0619 00:11:53.711067 25067 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 29 ms (attempt = 1)
I0619 00:11:53.680938 25796 raft_consensus.cc:479] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.713629 25796 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.715183 25796 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.715884 25741 tablet_replica.cc:261] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:53.716128 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:53.716331 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:53.716524 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:53.716742 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:53.716959 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:53.717164 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:53.717355 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(70b9c8c4af0945b69b5577e47b61c054)
I0619 00:11:53.717527 25741 raft_consensus.cc:2042] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:53.701972 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 186) took 1606ms. Request Metrics: {"tablet-delete.queue_time_us":1479033}
I0619 00:11:53.717752 25741 consensus_peers.cc:456] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:53.717962 25741 consensus_peers.cc:456] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
I0619 00:11:53.718425 25741 pending_rounds.cc:63] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Trying to abort 1 pending transactions.
I0619 00:11:53.718349 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 189) took 1569ms. Request Metrics: {"tablet-delete.queue_time_us":1569019}
I0619 00:11:53.718912 25325 ts_tablet_manager.cc:1173] T f75b8a4f1fd845fbab6f83f59b7cfc58 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.718750 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:53.719625 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "075076aaa16e4ea8a596a7a8c52f103e" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.719050 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 190) took 1566ms. Request Metrics: {"tablet-delete.queue_time_us":1565428}
I0619 00:11:53.720705 25623 raft_consensus.cc:2175] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
W0619 00:11:53.721812 25419 leader_election.cc:306] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.721897 25290 leader_election.cc:393] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:53.722235 25290 leader_election.cc:270] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.718578 25741 pending_rounds.cc:68] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390912327331840 op_type: NO_OP noop_request { }
I0619 00:11:53.722923 25741 raft_consensus.cc:2622] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:53.723146 25741 raft_consensus.cc:2069] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.723503 25724 ts_tablet_manager.cc:1028] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:53.723912 25724 tablet_bootstrap.cc:427] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:53.723799 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 191) took 1542ms. Request Metrics: {"tablet-delete.queue_time_us":1541234}
I0619 00:11:53.724305 25940 raft_consensus.cc:2732] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.663s (election complete)
W0619 00:11:53.727943 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.729373 25940 raft_consensus.cc:2784] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.729796 25724 tablet_bootstrap.cc:578] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:53.736587 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 58 ms (attempt = 2)
W0619 00:11:53.736888 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c has no consensus state
I0619 00:11:53.738812 25086 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:53.739665 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:53.742039 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:53.742801 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet f75b8a4f1fd845fbab6f83f59b7cfc58
I0619 00:11:53.743086 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.743444 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.743877 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.744503 25323 ts_tablet_manager.cc:1173] T 8e3a356933e749b9a4ade476ec30f475 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.727501 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 192) took 1547ms. Request Metrics: {"tablet-delete.queue_time_us":1547208}
I0619 00:11:53.746253 25330 tablet_service.cc:800] Processing DeleteTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.754952 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC with a delay of 77 ms (attempt = 3)
W0619 00:11:53.755244 25071 catalog_manager.cc:2910] Async tablet task fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fc7686daae274e0dba87703fe6924805 consensus state has no leader
I0619 00:11:53.755883 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) successfully deleted
W0619 00:11:53.756737 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.756731 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet 3537c79c30a1401e8a78b387f08eff58 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:53.757130 25602 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.757388 25598 tablet_service.cc:800] Processing DeleteTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.709239 25730 raft_consensus.cc:2042] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.757625 25601 tablet_service.cc:759] Processing CreateTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 (table=default.table_fa53e172a2a34136a5a6efb8b5c39368 [id=779e902dec1a4701a176d545a35e9978]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.758083 25600 tablet_service.cc:759] Processing CreateTablet for tablet 8e3a356933e749b9a4ade476ec30f475 (table=default.table_a822c7bace2a4cfd9c1c7f0a9c992551 [id=0d90b232db8e48ed9bfab0c59c6fbdba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.758252 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f75b8a4f1fd845fbab6f83f59b7cfc58. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.758664 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8e3a356933e749b9a4ade476ec30f475. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.763022 25885 raft_consensus.cc:457] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.763268 25885 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.538s (starting election)
I0619 00:11:53.763242 25325 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.763737 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet e3708e05f6054306b5bf7f1df2140e3d (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) successfully deleted
I0619 00:11:53.763510 25885 raft_consensus.cc:479] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.764164 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:53.764534 25885 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.764813 25730 raft_consensus.cc:2069] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.764976 25885 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:53.750535 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.766518 25730 ts_tablet_manager.cc:1369] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.752701 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775f0b6b309c49278066a5e7cf623af2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:53.753523 25423 consensus_peers.cc:434] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 397ae10bb6d841b4ba561ec431d29864. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.762142 25725 tablet_bootstrap.cc:427] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:53.767781 25725 ts_tablet_manager.cc:1045] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.097s	user 0.005s	sys 0.007s
I0619 00:11:53.767984 25601 ts_tablet_manager.cc:1173] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.767902 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4c653a255984d3bac88df63362b9334" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:53.769384 25553 leader_election.cc:306] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.769312 25725 raft_consensus.cc:325] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.769763 25083 master_service.cc:523] ReplaceTablet: received request to replace tablet e9a221a1d9814ef99a0b4ccaa04d1017 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:53.769966 25725 raft_consensus.cc:351] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.770174 25725 raft_consensus.cc:628] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:53.770495 25741 ts_tablet_manager.cc:1369] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.770604 25725 consensus_queue.cc:227] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.772794 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 03522c48c50943b38cae5aa88562e583 because the tablet was not found. No further retry: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
W0619 00:11:53.773228 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet edfdedbd4eca4b62be24b1649f820d0e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet edfdedbd4eca4b62be24b1649f820d0e already in progress: creating tablet
I0619 00:11:53.773449 25071 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 32 ms (attempt = 1)
I0619 00:11:53.778980 25940 raft_consensus.cc:2487] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.780277 25725 ts_tablet_manager.cc:1073] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I0619 00:11:53.780309 25727 tablet_bootstrap.cc:427] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:53.780658 25725 ts_tablet_manager.cc:1028] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:53.780691 25727 ts_tablet_manager.cc:1045] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.216s	user 0.005s	sys 0.006s
I0619 00:11:53.781004 25725 tablet_bootstrap.cc:427] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:53.782136 25727 raft_consensus.cc:325] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.782754 25727 raft_consensus.cc:351] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.782950 25727 raft_consensus.cc:628] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:53.784155 25727 consensus_queue.cc:227] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.784692 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:53.785132 25623 raft_consensus.cc:2784] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:53.785364 25623 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.642s (vote granted)
W0619 00:11:53.786576 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.785869 25468 tablet_service.cc:759] Processing CreateTablet for tablet 8e3a356933e749b9a4ade476ec30f475 (table=default.table_a822c7bace2a4cfd9c1c7f0a9c992551 [id=0d90b232db8e48ed9bfab0c59c6fbdba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.787132 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8e3a356933e749b9a4ade476ec30f475. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.787564 25727 ts_tablet_manager.cc:1073] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
W0619 00:11:53.788082 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.788801 25727 ts_tablet_manager.cc:1028] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:53.789202 25727 tablet_bootstrap.cc:427] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:53.790338 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet edfdedbd4eca4b62be24b1649f820d0e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet edfdedbd4eca4b62be24b1649f820d0e already in progress: creating tablet
I0619 00:11:53.790644 25071 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 56 ms (attempt = 1)
I0619 00:11:53.792412 25724 tablet_bootstrap.cc:427] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
W0619 00:11:53.792793 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.793197 25724 ts_tablet_manager.cc:1045] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.069s	user 0.002s	sys 0.003s
I0619 00:11:53.793176 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_970d5da055774f58881ea62e91e2f1f4" } new_table_name: "default.table_1ccfb0785fb045859278f19e637bbdc7" alter_external_catalogs: true
I0619 00:11:53.794648 25943 tablet.cc:1200] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:53.795292 25086 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3f22e2fb4a9e47aca16e414746eecd63" }
I0619 00:11:53.795684 25724 raft_consensus.cc:325] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.796098 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.796051 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 98 ms (attempt = 3)
W0619 00:11:53.796334 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c has no consensus state
I0619 00:11:53.796339 25724 raft_consensus.cc:351] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.796644 25724 raft_consensus.cc:628] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:53.797206 25724 consensus_queue.cc:227] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.798130 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_f05cbe4e96c04214a2f1a82f57ee9b77"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.798992 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f05cbe4e96c04214a2f1a82f57ee9b77 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:53.801833 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.803511 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.804179 25724 ts_tablet_manager.cc:1073] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.001s
I0619 00:11:53.804539 25724 ts_tablet_manager.cc:1028] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:53.804934 25724 tablet_bootstrap.cc:427] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:53.805953 25354 raft_consensus.cc:1081] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:53.807054 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:53.807323 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4c653a255984d3bac88df63362b9334" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.807984 25601 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.808995 25470 tablet_service.cc:759] Processing CreateTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 (table=default.table_fa53e172a2a34136a5a6efb8b5c39368 [id=779e902dec1a4701a176d545a35e9978]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.809645 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f75b8a4f1fd845fbab6f83f59b7cfc58. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:53.809964 25558 leader_election.cc:306] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.810205 25558 leader_election.cc:270] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.810936 25885 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.548s (election complete)
I0619 00:11:53.811156 25885 raft_consensus.cc:2487] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.814069 25082 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:53.815732 25796 consensus_queue.cc:963] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:53.816025 25082 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:53.816862 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:53.818768 25325 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.822098 25730 ts_tablet_manager.cc:1382] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.822372 25730 log.cc:986] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/4a88e394b199431e8b794a1b1aec847f
W0619 00:11:53.824043 25423 leader_election.cc:306] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:53.824323 25423 leader_election.cc:270] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.824687 25085 catalog_manager.cc:3663] Got report from deleted tablet d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:53.824810 25796 raft_consensus.cc:2732] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.886s (election complete)
I0619 00:11:53.825026 25796 raft_consensus.cc:2487] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:53.825139 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.825317 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:53.826797 25467 tablet_service.cc:800] Processing DeleteTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:53.827528 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.827530 25082 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:53.828150 25082 catalog_manager.cc:3663] Got report from deleted tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:53.828291 25601 tablet_service.cc:800] Processing DeleteTablet for tablet d62715e230154e2abe126ef97feec7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.828462 25082 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:53.828771 25082 catalog_manager.cc:3663] Got report from deleted tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:53.828830 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 193) took 1646ms. Request Metrics: {"tablet-delete.queue_time_us":1645716}
I0619 00:11:53.829499 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 196) took 1601ms. Request Metrics: {"tablet-delete.queue_time_us":1601089}
I0619 00:11:53.830072 25736 tablet_replica.cc:261] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:53.830222 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:53.830318 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:53.830546 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:53.830566 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:53.830773 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:53.830960 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:53.830936 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:53.831187 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:53.831276 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:53.831392 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(a52cbcbe35374d518bca1e0fc24d88d4)
W0619 00:11:53.832093 25558 leader_election.cc:306] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.832360 25558 leader_election.cc:270] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.832830 25885 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.809s (election complete)
I0619 00:11:53.833077 25885 raft_consensus.cc:2487] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.834028 25741 ts_tablet_manager.cc:1382] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:53.834283 25741 log.cc:986] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/70b9c8c4af0945b69b5577e47b61c054
I0619 00:11:53.834857 25741 ts_tablet_manager.cc:1403] T 70b9c8c4af0945b69b5577e47b61c054 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:53.841462 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC with a delay of 172 ms (attempt = 4)
W0619 00:11:53.841737 25071 catalog_manager.cc:2910] Async tablet task fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fc7686daae274e0dba87703fe6924805 consensus state has no leader
I0619 00:11:53.791404 25727 tablet_bootstrap.cc:578] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:53.845273 25725 tablet_bootstrap.cc:578] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:53.848944 25600 ts_tablet_manager.cc:1173] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.849112 25068 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC with a delay of 1033 ms (attempt = 7)
W0619 00:11:53.849369 25068 catalog_manager.cc:2910] Async tablet task 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
I0619 00:11:53.849684 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:53.849704 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 548 ms (attempt = 6)
W0619 00:11:53.850033 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:53.850790 25885 raft_consensus.cc:457] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.850993 25885 raft_consensus.cc:2732] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.045s (starting election)
I0619 00:11:53.851205 25885 raft_consensus.cc:479] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.852097 25885 leader_election.cc:243] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.852448 25885 leader_election.cc:243] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.852854 25724 tablet_bootstrap.cc:578] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:53.853049 25730 ts_tablet_manager.cc:1403] T 4a88e394b199431e8b794a1b1aec847f P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:53.854528 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0474be659e5e4a4cbfc181b8c812b1eb" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:53.855953 25553 leader_election.cc:306] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.856534 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet e9a221a1d9814ef99a0b4ccaa04d1017
I0619 00:11:53.856845 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:11:53.857221 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:11:53.857611 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:11:53.857802 25468 ts_tablet_manager.cc:1173] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.857906 25083 catalog_manager.cc:4596] ReplaceTablet: tablet e9a221a1d9814ef99a0b4ccaa04d1017 deleted and replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1
I0619 00:11:53.831574 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:53.858126 25736 raft_consensus.cc:2042] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.858415 25736 raft_consensus.cc:2069] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.859627 25325 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.861227 25087 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 42 ms (attempt = 1)
W0619 00:11:53.861497 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:53.862048 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 93930ffe285d48d080a8ca4793e871d9
I0619 00:11:53.862284 25600 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.862336 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.862730 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.863117 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:53.863982 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.864379 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:53.872855 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.875751 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.876806 25942 tablet.cc:1195] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
W0619 00:11:53.878756 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.880331 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 148) took 1607ms. Request Metrics: {"tablet-delete.queue_time_us":1432257}
I0619 00:11:53.881079 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 150) took 1531ms. Request Metrics: {"tablet-delete.queue_time_us":1515811}
I0619 00:11:53.881654 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 149) took 1532ms. Request Metrics: {"tablet-delete.queue_time_us":1516099}
I0619 00:11:53.881844 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) successfully deleted
W0619 00:11:53.882331 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:53.882248 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 153) took 1385ms. Request Metrics: {"tablet-delete.queue_time_us":1385080}
W0619 00:11:53.882777 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:53.882985 25623 raft_consensus.cc:2245] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
W0619 00:11:53.883266 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.883579 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 154) took 1385ms. Request Metrics: {"tablet-delete.queue_time_us":1383614}
I0619 00:11:53.884155 25730 tablet_replica.cc:261] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:53.884385 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:53.884578 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:53.884783 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:53.884974 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:53.885186 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:53.885381 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:53.885558 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:53.885746 25730 raft_consensus.cc:2042] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
W0619 00:11:53.885835 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 because the tablet was not found. No further retry: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:53.886015 25730 raft_consensus.cc:2069] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.886369 25942 tablet.cc:1195] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
I0619 00:11:53.887590 25730 ts_tablet_manager.cc:1369] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:53.888604 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:53.890038 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.891854 25623 raft_consensus.cc:1081] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:53.894402 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 212) took 1682ms. Request Metrics: {"spinlock_wait_cycles":2816,"tablet-delete.queue_time_us":1496514}
I0619 00:11:53.895031 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 149 ms (attempt = 4)
I0619 00:11:53.895145 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 214) took 1676ms. Request Metrics: {"tablet-delete.queue_time_us":1644014}
W0619 00:11:53.895347 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c has no consensus state
I0619 00:11:53.895429 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 70b9c8c4af0945b69b5577e47b61c054 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) successfully deleted
I0619 00:11:53.896566 25736 ts_tablet_manager.cc:1369] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:53.897786 25470 ts_tablet_manager.cc:1173] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:53.898718 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0474be659e5e4a4cbfc181b8c812b1eb" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.899503 25899 tablet.cc:1200] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:53.904685 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 41 ms (attempt = 2)
W0619 00:11:53.904948 25067 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
W0619 00:11:53.907933 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet f372b9fb174844eca1662c62039a3c71 because the tablet was not found. No further retry: Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
W0619 00:11:53.908287 25419 consensus_peers.cc:434] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 30a9d9fda8d34caeb8242c372439321b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:11:53.908427 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet f372b9fb174844eca1662c62039a3c71 because the tablet was not found. No further retry: Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
I0619 00:11:53.901988 25943 tablet.cc:1200] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
W0619 00:11:53.908860 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a4c653a255984d3bac88df63362b9334 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a4c653a255984d3bac88df63362b9334 already in progress: creating tablet
I0619 00:11:53.909097 25071 catalog_manager.cc:2891] Scheduling retry of a4c653a255984d3bac88df63362b9334 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 70 ms (attempt = 2)
I0619 00:11:53.909037 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 215) took 1639ms. Request Metrics: {"tablet-delete.queue_time_us":1637435}
I0619 00:11:53.909512 25423 leader_election.cc:379] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:53.909425 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 70b9c8c4af0945b69b5577e47b61c054 because the tablet was not found. No further retry: Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054
W0619 00:11:53.915838 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4a88e394b199431e8b794a1b1aec847f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 4a88e394b199431e8b794a1b1aec847f already in progress: creating tablet
I0619 00:11:53.916085 25067 catalog_manager.cc:2891] Scheduling retry of 4a88e394b199431e8b794a1b1aec847f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 16 ms (attempt = 1)
I0619 00:11:53.917865 25423 leader_election.cc:270] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:53.913511 25725 tablet_bootstrap.cc:427] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:53.918280 25725 ts_tablet_manager.cc:1045] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.137s	user 0.004s	sys 0.003s
I0619 00:11:53.919574 25725 raft_consensus.cc:325] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.920420 25725 raft_consensus.cc:351] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.920745 25725 raft_consensus.cc:628] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:53.921190 25725 consensus_queue.cc:227] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:53.921764 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.915436 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 218) took 1593ms. Request Metrics: {"tablet-delete.queue_time_us":1591893}
I0619 00:11:53.923653 25725 ts_tablet_manager.cc:1073] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:11:53.924000 25725 ts_tablet_manager.cc:1028] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:53.923213 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 219) took 1547ms. Request Metrics: {"tablet-delete.queue_time_us":1545169}
I0619 00:11:53.928728 25885 raft_consensus.cc:457] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.928993 25885 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.972s (starting election)
I0619 00:11:53.929270 25885 raft_consensus.cc:479] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.930261 25885 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:53.930668 25885 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.932507 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_f749a3c527f5449f8d30ae159bf2a2cb" } new_table_name: "default.table_b4f172b08ffd4adb9f52e785f33e3076" alter_external_catalogs: true
I0619 00:11:53.929394 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 220) took 1541ms. Request Metrics: {"tablet-delete.queue_time_us":1540623}
W0619 00:11:53.934788 25290 consensus_peers.cc:434] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet a85577b0beba4276b963e5c360544bc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:53.935817 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:53.937121 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:53.937207 25553 leader_election.cc:306] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:53.937942 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a4c653a255984d3bac88df63362b9334 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a4c653a255984d3bac88df63362b9334 already in progress: creating tablet
I0619 00:11:53.938153 25067 catalog_manager.cc:2891] Scheduling retry of a4c653a255984d3bac88df63362b9334 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 66 ms (attempt = 2)
I0619 00:11:53.939702 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.934167 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 222) took 1480ms. Request Metrics: {"tablet-delete.queue_time_us":1479372}
W0619 00:11:53.941912 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
I0619 00:11:53.942108 25724 tablet_bootstrap.cc:427] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:53.942443 25724 ts_tablet_manager.cc:1045] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.138s	user 0.007s	sys 0.007s
W0619 00:11:53.942821 25558 leader_election.cc:306] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:53.943099 25558 leader_election.cc:270] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.942962 25730 ts_tablet_manager.cc:1382] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.943753 25885 raft_consensus.cc:2732] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.601s (election complete)
I0619 00:11:53.944350 25885 raft_consensus.cc:2487] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.944718 25730 log.cc:986] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a4c653a255984d3bac88df63362b9334
I0619 00:11:53.945260 25623 raft_consensus.cc:1081] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:53.941452 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 223) took 1475ms. Request Metrics: {"tablet-delete.queue_time_us":1474571}
I0619 00:11:53.946864 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 103 ms (attempt = 3)
I0619 00:11:53.947060 25741 tablet_replica.cc:261] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
W0619 00:11:53.947118 25067 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:53.947291 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(c18355df78d344839a56189c7549ab1f)
W0619 00:11:53.947464 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7f60ea5c5def47ee846d0186bbc90c18 already in progress: creating tablet
I0619 00:11:53.947500 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:53.947711 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:53.947695 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 168 ms (attempt = 4)
I0619 00:11:53.947899 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:53.924432 25847 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.642s (election complete)
I0619 00:11:53.948125 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:53.948328 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:53.948268 25847 raft_consensus.cc:2539] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:53.948513 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(c18355df78d344839a56189c7549ab1f)
I0619 00:11:53.948581 25847 raft_consensus.cc:593] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:53.948711 25741 raft_consensus.cc:2042] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:53.948977 25741 raft_consensus.cc:2069] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:53.949230 25730 ts_tablet_manager.cc:1403] T a4c653a255984d3bac88df63362b9334 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:53.949046 25847 consensus_queue.cc:204] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.950202 25741 ts_tablet_manager.cc:1369] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:53.925308 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.952297 25940 raft_consensus.cc:457] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:53.952366 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 3537c79c30a1401e8a78b387f08eff58
I0619 00:11:53.952549 25940 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.076s (starting election)
I0619 00:11:53.952689 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:53.953056 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:53.952824 25940 raft_consensus.cc:479] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.953413 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:53.953729 25080 catalog_manager.cc:4596] ReplaceTablet: tablet 3537c79c30a1401e8a78b387f08eff58 deleted and replaced by tablet d0624684b6b04ba3be260b2705dc00e1
I0619 00:11:53.953819 25940 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:53.953963 25086 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 51 ms (attempt = 1)
W0619 00:11:53.954216 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f has no consensus state
I0619 00:11:53.954213 25940 leader_election.cc:243] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:53.956761 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.957156 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:53.957768 25623 raft_consensus.cc:2175] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:53.957653 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3537c79c30a1401e8a78b387f08eff58" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:53.958204 25487 raft_consensus.cc:2206] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:53.959223 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.943780 25724 raft_consensus.cc:325] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.959676 25724 raft_consensus.cc:351] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.959867 25724 raft_consensus.cc:628] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:53.959858 25290 leader_election.cc:393] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:53.960508 25290 leader_election.cc:393] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:53.960779 25290 leader_election.cc:270] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:53.961179 25940 raft_consensus.cc:2732] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.536s (election complete)
I0619 00:11:53.961031 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3e0e0119254a4be881b82ecf36174935"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:53.961416 25940 raft_consensus.cc:2784] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:53.961910 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3e0e0119254a4be881b82ecf36174935 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.960319 25724 consensus_queue.cc:227] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.966028 25940 raft_consensus.cc:2487] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:53.937278 25736 ts_tablet_manager.cc:1382] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:53.966410 25736 log.cc:986] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a52cbcbe35374d518bca1e0fc24d88d4
W0619 00:11:53.968386 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.967929 25736 ts_tablet_manager.cc:1403] T a52cbcbe35374d518bca1e0fc24d88d4 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:53.970036 25724 ts_tablet_manager.cc:1073] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.000s
I0619 00:11:53.970369 25724 ts_tablet_manager.cc:1028] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:53.971346 25724 tablet_bootstrap.cc:427] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:53.971792 25085 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
W0619 00:11:53.972112 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:53.972180 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:53.972347 25085 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:53.972782 25085 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:53.973038 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 155) took 1464ms. Request Metrics: {"tablet-delete.queue_time_us":1375369}
I0619 00:11:53.973734 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) successfully deleted
I0619 00:11:53.974234 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:53.974125 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 159) took 1363ms. Request Metrics: {"tablet-delete.queue_time_us":1362859}
I0619 00:11:53.974445 25725 tablet_bootstrap.cc:427] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:53.974848 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 981d862d1bab4b14bb10e6fc2174e1cc already in progress: creating tablet
I0619 00:11:53.975075 25071 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 16 ms (attempt = 1)
I0619 00:11:53.975443 25084 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:53.975888 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c24f1b02a5b54618aca5028342d84b7d" }
I0619 00:11:53.976569 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 160) took 1366ms. Request Metrics: {"tablet-delete.queue_time_us":1363558}
I0619 00:11:53.978276 25730 tablet_replica.cc:261] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
W0619 00:11:53.978574 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 54adb588955b45b79de9752f254728ea with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 54adb588955b45b79de9752f254728ea already in progress: creating tablet
I0619 00:11:53.978813 25071 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 47 ms (attempt = 1)
I0619 00:11:53.979063 25325 tablet_service.cc:759] Processing CreateTablet for tablet b50379774e4d4fdfb823ab2dc919db30 (table=default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.979637 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b50379774e4d4fdfb823ab2dc919db30. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.979763 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_e874b4b421a446c48acc51dc98a14038"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.980592 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e874b4b421a446c48acc51dc98a14038 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.981421 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:53.981514 25727 tablet_bootstrap.cc:427] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:53.981881 25727 ts_tablet_manager.cc:1045] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.193s	user 0.008s	sys 0.004s
I0619 00:11:53.982450 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_6ebe326fe1fe4f3580d3f06acc67821f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:53.983300 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6ebe326fe1fe4f3580d3f06acc67821f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:53.986076 25727 raft_consensus.cc:325] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:53.987521 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:53.987723 25727 raft_consensus.cc:351] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:53.974608 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:53.988140 25727 raft_consensus.cc:628] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:53.988432 25323 tablet_service.cc:759] Processing CreateTablet for tablet cbd67634f5704165b5bb948501697f0c (table=default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.989032 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cbd67634f5704165b5bb948501697f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:53.994035 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:53.994331 25419 consensus_peers.cc:434] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 30a9d9fda8d34caeb8242c372439321b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:53.989006 25727 consensus_queue.cc:227] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.000811 25727 ts_tablet_manager.cc:1073] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I0619 00:11:54.002254 25947 consensus_queue.cc:963] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:54.005506 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 1dd3f7afe05f49199b18220f9c11ccd2
I0619 00:11:54.005820 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:54.006213 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:54.006562 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.006659 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:54.006939 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:54.007095 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.007748 25071 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 58 ms (attempt = 2)
W0619 00:11:54.008018 25071 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f has no consensus state
I0619 00:11:54.009493 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.009680 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:54.010008 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.010206 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.010749 25330 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.015239 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC with a delay of 256 ms (attempt = 5)
W0619 00:11:54.015516 25071 catalog_manager.cc:2910] Async tablet task fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fc7686daae274e0dba87703fe6924805 consensus state has no leader
I0619 00:11:54.016921 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:53.982903 25601 tablet_service.cc:759] Processing CreateTablet for tablet cbd67634f5704165b5bb948501697f0c (table=default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.983889 25600 tablet_service.cc:759] Processing CreateTablet for tablet b50379774e4d4fdfb823ab2dc919db30 (table=default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:53.994452 25885 raft_consensus.cc:457] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.017321 25885 raft_consensus.cc:2732] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.826s (starting election)
I0619 00:11:54.017518 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cbd67634f5704165b5bb948501697f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.009023 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:53.995117 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.017616 25885 raft_consensus.cc:479] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.018533 25086 catalog_manager.cc:3663] Got report from deleted tablet 775f0b6b309c49278066a5e7cf623af2 (table default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:54.018857 25086 catalog_manager.cc:3663] Got report from deleted tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:54.018955 25885 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.019342 25885 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.020905 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b50379774e4d4fdfb823ab2dc919db30. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.024515 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.026654 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.026962 25601 ts_tablet_manager.cc:1173] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.028198 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:53.979398 25724 tablet_bootstrap.cc:578] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:54.030091 25600 ts_tablet_manager.cc:1173] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.032035 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.033663 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.033905 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.034109 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.034339 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.034631 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.034842 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.035014 25730 raft_consensus.cc:2042] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.035276 25730 raft_consensus.cc:2069] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.027842 25725 tablet_bootstrap.cc:578] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:54.037130 25727 ts_tablet_manager.cc:1028] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:54.037544 25727 tablet_bootstrap.cc:427] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:54.002584 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 200) took 1716ms. Request Metrics: {"mutex_wait_us":32496,"spinlock_wait_cycles":15521664,"tablet-delete.queue_time_us":1543830}
I0619 00:11:54.038627 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 201) took 1752ms. Request Metrics: {"tablet-delete.queue_time_us":1751880}
I0619 00:11:54.039213 25736 tablet_replica.cc:261] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:54.039450 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:54.039659 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:54.039886 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:54.040086 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:54.040323 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:54.040534 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:54.040755 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:54.040832 25727 tablet_bootstrap.cc:578] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:54.040932 25736 raft_consensus.cc:2042] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.041194 25736 raft_consensus.cc:2069] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.043045 25325 ts_tablet_manager.cc:1173] T b50379774e4d4fdfb823ab2dc919db30 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.045007 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d62715e230154e2abe126ef97feec7ce" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.045871 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 268 ms (attempt = 5)
I0619 00:11:54.045990 25354 raft_consensus.cc:1081] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:11:54.046135 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c consensus state has no leader
W0619 00:11:54.047617 25553 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.047473 25947 consensus_queue.cc:963] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:11:54.048491 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.048593 25086 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45] alter complete (version 1)
I0619 00:11:54.049350 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:54.049743 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:54.050487 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 156 ms (attempt = 4)
W0619 00:11:54.050793 25067 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:54.052588 25724 tablet_bootstrap.cc:427] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:54.052906 25323 ts_tablet_manager.cc:1173] T cbd67634f5704165b5bb948501697f0c P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.052940 25724 ts_tablet_manager.cc:1045] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.082s	user 0.005s	sys 0.000s
I0619 00:11:54.054589 25724 raft_consensus.cc:325] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.055241 25724 raft_consensus.cc:351] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.055435 25724 raft_consensus.cc:628] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:54.056984 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.057133 25622 raft_consensus.cc:1081] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:54.058235 25736 ts_tablet_manager.cc:1369] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.059087 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:54.067320 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.068231 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d62715e230154e2abe126ef97feec7ce" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:54.069555 25558 leader_election.cc:306] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.069856 25558 leader_election.cc:270] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.070340 25885 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.716s (election complete)
I0619 00:11:54.070555 25885 raft_consensus.cc:2487] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.071048 25470 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.071820 25947 consensus_queue.cc:963] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:54.072306 25725 tablet_bootstrap.cc:427] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:54.072809 25725 ts_tablet_manager.cc:1045] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.099s	user 0.006s	sys 0.000s
I0619 00:11:54.073253 25470 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:54.074599 25558 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.074904 25558 leader_election.cc:270] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.075409 25885 raft_consensus.cc:2732] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.930s (election complete)
I0619 00:11:54.075649 25885 raft_consensus.cc:2487] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.075980 25947 raft_consensus.cc:457] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.076292 25947 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.603s (starting election)
I0619 00:11:54.076515 25947 raft_consensus.cc:479] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.077467 25947 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.077870 25947 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.079334 25748 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:11:54.079121 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7f60ea5c5def47ee846d0186bbc90c18" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.076037 25725 raft_consensus.cc:325] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.080679 25419 leader_election.cc:306] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.080489 25725 raft_consensus.cc:351] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.082232 25730 ts_tablet_manager.cc:1369] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.082110 25725 raft_consensus.cc:628] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:54.088857 25290 consensus_peers.cc:434] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet a85577b0beba4276b963e5c360544bc7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:54.089224 25923 tablet.cc:1200] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
W0619 00:11:54.091621 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.092440 25069 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 1052 ms (attempt = 7)
I0619 00:11:54.092665 25085 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
W0619 00:11:54.092754 25069 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:54.092988 25085 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:54.094110 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:54.097455 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 1 got 0
I0619 00:11:54.100517 25947 consensus_queue.cc:963] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:54.100847 25748 master_main.cc:83] Starting Master server...
I0619 00:11:54.104190 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:54.104658 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.096113 25724 consensus_queue.cc:227] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.108872 25724 ts_tablet_manager.cc:1073] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.056s	user 0.005s	sys 0.000s
I0619 00:11:54.109206 25724 ts_tablet_manager.cc:1028] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:54.087729 25725 consensus_queue.cc:227] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.109338 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.110507 25724 tablet_bootstrap.cc:427] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:54.111186 25727 tablet_bootstrap.cc:427] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:54.111554 25727 ts_tablet_manager.cc:1045] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.074s	user 0.002s	sys 0.004s
I0619 00:11:54.111694 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.112962 25727 raft_consensus.cc:325] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.114814 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.115914 25086 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:54.116524 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:54.117730 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.121757 25084 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:54.121841 25724 tablet_bootstrap.cc:578] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:54.122412 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:54.123319 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
W0619 00:11:54.129382 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.132234 25730 ts_tablet_manager.cc:1382] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I0619 00:11:54.132716 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 2062 ms (attempt = 8)
W0619 00:11:54.133033 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.133404 25071 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 93 ms (attempt = 3)
W0619 00:11:54.133698 25071 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
W0619 00:11:54.134120 25071 catalog_manager.cc:2910] Async tablet task acc89522b693498c987ca5d8ef8275ea (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet acc89522b693498c987ca5d8ef8275ea consensus state has no leader
I0619 00:11:54.134413 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) successfully deleted
I0619 00:11:54.134994 25736 ts_tablet_manager.cc:1382] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:54.135303 25736 log.cc:986] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a85577b0beba4276b963e5c360544bc7
I0619 00:11:54.135262 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_b4f172b08ffd4adb9f52e785f33e3076" } new_table_name: "default.table_3186954a4b3d40d8be527ce49a270d8b" alter_external_catalogs: true
I0619 00:11:54.135926 25736 ts_tablet_manager.cc:1403] T a85577b0beba4276b963e5c360544bc7 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:54.137161 25725 ts_tablet_manager.cc:1073] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.064s	user 0.004s	sys 0.000s
W0619 00:11:54.137219 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times.
W0619 00:11:54.118222 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.118851 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.123330 25325 tablet_service.cc:759] Processing CreateTablet for tablet ef1566aa89b24620af70f2322df0dd56 (table=default.table_f05cbe4e96c04214a2f1a82f57ee9b77 [id=0c557a51d7e24b90ac96183eda7f4785]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.138487 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ef1566aa89b24620af70f2322df0dd56. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:54.137903 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:54.123965 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.141240 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d036ca047e034c39958950ab500d6004"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:54.129963 25336 tablet_service.cc:759] Processing CreateTablet for tablet 31b36efa3dbd4aa983b3a396aa9d62f1 (table=default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.142127 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 31b36efa3dbd4aa983b3a396aa9d62f1. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:54.142128 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d036ca047e034c39958950ab500d6004 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.144151 25083 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_4c17e965f551446d965e5fa1499cad9a" }
I0619 00:11:54.145812 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 2056 ms (attempt = 8)
W0619 00:11:54.146075 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:54.147980 25085 master_service.cc:523] ReplaceTablet: received request to replace tablet e60ecc51c92c461889b360dde1dc5cf2 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:54.153978 25330 tablet_service.cc:759] Processing CreateTablet for tablet b8bc8d4a970843258ab5e9530f425d0f (table=default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.154575 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b8bc8d4a970843258ab5e9530f425d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.157143 25727 raft_consensus.cc:351] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.157353 25727 raft_consensus.cc:628] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:54.158048 25727 consensus_queue.cc:227] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.159976 25086 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 19 ms (attempt = 1)
W0619 00:11:54.160262 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
W0619 00:11:54.162351 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.166528 25086 catalog_manager.cc:3663] Got report from deleted tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:54.167047 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet ee4c4769f75141f9b2ef864ab8a8d9e0
I0619 00:11:54.167349 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.167609 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:54.167767 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.168190 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.169845 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet e60ecc51c92c461889b360dde1dc5cf2
I0619 00:11:54.170104 25323 tablet_service.cc:800] Processing DeleteTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.170181 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:54.170547 25331 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.170655 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:54.171109 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:54.171281 25331 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.171442 25085 catalog_manager.cc:4596] ReplaceTablet: tablet e60ecc51c92c461889b360dde1dc5cf2 deleted and replaced by tablet 28f9fefa25d14aa089c34c9a0825946e
I0619 00:11:54.128352 25741 ts_tablet_manager.cc:1382] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:54.172278 25741 log.cc:986] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/c18355df78d344839a56189c7549ab1f
I0619 00:11:54.172823 25741 ts_tablet_manager.cc:1403] T c18355df78d344839a56189c7549ab1f P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:54.173363 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 224) took 1706ms. Request Metrics: {"tablet-delete.queue_time_us":1479908}
I0619 00:11:54.174459 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 225) took 1697ms. Request Metrics: {"tablet-delete.queue_time_us":1697140}
I0619 00:11:54.175498 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 226) took 1676ms. Request Metrics: {"tablet-delete.queue_time_us":1675506}
I0619 00:11:54.176434 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 230) took 1636ms. Request Metrics: {"tablet-delete.queue_time_us":1635663}
I0619 00:11:54.177263 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
I0619 00:11:54.178349 25336 ts_tablet_manager.cc:1173] T 31b36efa3dbd4aa983b3a396aa9d62f1 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:54.150743 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.151348 25947 raft_consensus.cc:457] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.179632 25947 raft_consensus.cc:2732] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.192s (starting election)
I0619 00:11:54.179921 25947 raft_consensus.cc:479] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.180933 25947 leader_election.cc:243] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.181339 25947 leader_election.cc:243] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.179409 25725 ts_tablet_manager.cc:1028] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:54.153070 25470 tablet_service.cc:759] Processing CreateTablet for tablet b50379774e4d4fdfb823ab2dc919db30 (table=default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.181921 25725 tablet_bootstrap.cc:427] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:54.182278 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b50379774e4d4fdfb823ab2dc919db30. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.182467 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4a88e394b199431e8b794a1b1aec847f" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:54.183943 25419 leader_election.cc:306] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:54.185397 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.187665 25330 ts_tablet_manager.cc:1173] T b8bc8d4a970843258ab5e9530f425d0f P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.140296 25600 tablet_service.cc:759] Processing CreateTablet for tablet ef1566aa89b24620af70f2322df0dd56 (table=default.table_f05cbe4e96c04214a2f1a82f57ee9b77 [id=0c557a51d7e24b90ac96183eda7f4785]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.140312 25601 tablet_service.cc:759] Processing CreateTablet for tablet 31b36efa3dbd4aa983b3a396aa9d62f1 (table=default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.188735 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ef1566aa89b24620af70f2322df0dd56. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:54.180887 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.189354 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 31b36efa3dbd4aa983b3a396aa9d62f1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.189723 25727 ts_tablet_manager.cc:1073] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.078s	user 0.004s	sys 0.000s
I0619 00:11:54.190055 25727 ts_tablet_manager.cc:1028] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:54.190374 25727 tablet_bootstrap.cc:427] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:54.191493 25470 ts_tablet_manager.cc:1173] T b50379774e4d4fdfb823ab2dc919db30 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.184015 25885 raft_consensus.cc:457] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.191781 25885 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.553s (starting election)
W0619 00:11:54.153594 25423 leader_election.cc:306] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:54.192159 25423 leader_election.cc:270] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.192024 25885 raft_consensus.cc:479] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.192827 25947 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.954s (election complete)
I0619 00:11:54.192940 25885 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.193048 25947 raft_consensus.cc:2487] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.193322 25885 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.194367 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:54.195672 25553 leader_election.cc:306] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.178248 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet c18355df78d344839a56189c7549ab1f (table default.table_50bf60a48eb949a2b7c22f6b61030f66 [id=3e2a135384014c7b9626f9a8df3162aa]) successfully deleted
I0619 00:11:54.197593 25067 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 66 ms (attempt = 2)
W0619 00:11:54.197877 25067 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
W0619 00:11:54.198343 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
W0619 00:11:54.199290 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 981d862d1bab4b14bb10e6fc2174e1cc already in progress: creating tablet
I0619 00:11:54.199501 25067 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 35 ms (attempt = 1)
I0619 00:11:54.199610 25601 ts_tablet_manager.cc:1173] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.199671 25325 ts_tablet_manager.cc:1173] T ef1566aa89b24620af70f2322df0dd56 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.200816 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 1035 ms (attempt = 7)
I0619 00:11:54.200809 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 231) took 1655ms. Request Metrics: {"tablet-delete.queue_time_us":1654877}
W0619 00:11:54.201164 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_34ae54d8d2fc4703a6fd963ddd934b4d [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:54.201236 25730 log.cc:986] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/36f5eec972454080b7b434b4df7e62e3
W0619 00:11:54.201552 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 54adb588955b45b79de9752f254728ea with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 54adb588955b45b79de9752f254728ea already in progress: creating tablet
I0619 00:11:54.201817 25067 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
I0619 00:11:54.202355 25741 tablet_replica.cc:261] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:54.202489 25083 catalog_manager.cc:3663] Got report from deleted tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:54.192553 25727 tablet_bootstrap.cc:578] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:54.203169 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a52cbcbe35374d518bca1e0fc24d88d4 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:54.202649 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.203498 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.203768 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.204000 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.203809 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_50bf60a48eb949a2b7c22f6b61030f66" } new_table_name: "default.table_951214baec60486e8b27aec35a5923e6" alter_external_catalogs: true
I0619 00:11:54.204279 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.204560 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.204823 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.205034 25741 raft_consensus.cc:2042] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.205149 25083 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:54.205330 25741 raft_consensus.cc:2069] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.205826 25730 ts_tablet_manager.cc:1403] T 36f5eec972454080b7b434b4df7e62e3 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:54.205799 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:54.206817 25598 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.208106 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 161) took 1552ms. Request Metrics: {"tablet-delete.queue_time_us":1322295}
I0619 00:11:54.208557 25602 tablet_service.cc:759] Processing CreateTablet for tablet b8bc8d4a970843258ab5e9530f425d0f (table=default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.209172 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b8bc8d4a970843258ab5e9530f425d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.210654 25601 tablet_service.cc:800] Processing DeleteTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.213685 25084 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 21 ms (attempt = 1)
W0619 00:11:54.213950 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.215013 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 292 ms (attempt = 5)
W0619 00:11:54.215313 25067 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:54.216966 25084 master_service.cc:523] ReplaceTablet: received request to replace tablet 075076aaa16e4ea8a596a7a8c52f103e from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:54.218410 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 166) took 1359ms. Request Metrics: {"tablet-delete.queue_time_us":1358938}
I0619 00:11:54.219359 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 167) took 1358ms. Request Metrics: {"tablet-delete.queue_time_us":1357470}
I0619 00:11:54.220315 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 165) took 1362ms. Request Metrics: {"tablet-delete.queue_time_us":1343126}
I0619 00:11:54.221191 25325 tablet_service.cc:800] Processing DeleteTablet for tablet a52cbcbe35374d518bca1e0fc24d88d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.221877 25724 tablet_bootstrap.cc:427] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:54.222414 25325 tablet_service.cc:759] Processing CreateTablet for tablet d0624684b6b04ba3be260b2705dc00e1 (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.222537 25724 ts_tablet_manager.cc:1045] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.112s	user 0.003s	sys 0.003s
I0619 00:11:54.223026 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d0624684b6b04ba3be260b2705dc00e1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.223943 25724 raft_consensus.cc:325] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.224609 25724 raft_consensus.cc:351] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.224848 25724 raft_consensus.cc:628] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:54.225402 25724 consensus_queue.cc:227] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.227319 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 075076aaa16e4ea8a596a7a8c52f103e
I0619 00:11:54.227756 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
W0619 00:11:54.229454 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.232190 25083 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:54.232905 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:54.234750 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.236531 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.238534 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
I0619 00:11:54.238963 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
W0619 00:11:54.239034 25290 consensus_peers.cc:434] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet cb57d196f0724247b6a7f9a34be8377a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:54.239267 25084 catalog_manager.cc:4596] ReplaceTablet: tablet 075076aaa16e4ea8a596a7a8c52f103e deleted and replaced by tablet dba1210d9c22454f8c9e0b0293583f0c
I0619 00:11:54.239590 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 74 ms (attempt = 2)
W0619 00:11:54.239928 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.242924 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:54.244649 25336 tablet_service.cc:759] Processing CreateTablet for tablet 756057a7c34a4c61a261deaf849e433a (table=default.table_252d88d7d98b495e80affd5dc6eeb739 [id=f8e9404177f44b2cba84c5146b780133]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.245239 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 756057a7c34a4c61a261deaf849e433a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.248392 25725 tablet_bootstrap.cc:578] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:54.249155 25600 ts_tablet_manager.cc:1173] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.249830 25083 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:54.250468 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:54.252944 25602 ts_tablet_manager.cc:1173] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.252910 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 203) took 1966ms. Request Metrics: {"tablet-delete.queue_time_us":1752268}
I0619 00:11:54.253659 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.254426 25071 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 162 ms (attempt = 4)
I0619 00:11:54.254443 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 204) took 1967ms. Request Metrics: {"tablet-delete.queue_time_us":1967073}
W0619 00:11:54.254725 25071 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:54.254822 25741 ts_tablet_manager.cc:1369] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.254990 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 36f5eec972454080b7b434b4df7e62e3 (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) successfully deleted
I0619 00:11:54.255057 25736 tablet_replica.cc:261] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:54.255295 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:54.255511 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:54.255741 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:54.255955 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(4a88e394b199431e8b794a1b1aec847f)
W0619 00:11:54.256081 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 4a88e394b199431e8b794a1b1aec847f because the tablet was not found. No further retry: Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:54.256198 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:54.256408 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:54.256664 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:54.256898 25736 raft_consensus.cc:2042] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.257263 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.257457 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.258298 25724 ts_tablet_manager.cc:1073] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.000s
I0619 00:11:54.258363 25598 tablet_service.cc:759] Processing CreateTablet for tablet 756057a7c34a4c61a261deaf849e433a (table=default.table_252d88d7d98b495e80affd5dc6eeb739 [id=f8e9404177f44b2cba84c5146b780133]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.258440 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:54.258800 25724 ts_tablet_manager.cc:1028] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:54.259011 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 756057a7c34a4c61a261deaf849e433a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.259225 25724 tablet_bootstrap.cc:427] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:54.259178 25084 catalog_manager.cc:3663] Got report from deleted tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:54.259837 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:54.259861 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.261807 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.262486 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.263195 25730 tablet_replica.cc:261] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:54.263705 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.263968 25602 tablet_service.cc:759] Processing CreateTablet for tablet d0624684b6b04ba3be260b2705dc00e1 (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.264540 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d0624684b6b04ba3be260b2705dc00e1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.264159 25467 tablet_service.cc:759] Processing CreateTablet for tablet cbd67634f5704165b5bb948501697f0c (table=default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.265507 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cbd67634f5704165b5bb948501697f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.267804 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:54.267905 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.268872 25067 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 86 ms (attempt = 3)
W0619 00:11:54.269925 25067 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:54.270167 25727 tablet_bootstrap.cc:427] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:54.270515 25727 ts_tablet_manager.cc:1045] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.080s	user 0.005s	sys 0.000s
I0619 00:11:54.271103 25331 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.263103 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_53e73552aa6d4161a595d779a9fc96cc"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:54.272295 25727 raft_consensus.cc:325] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.273072 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_53e73552aa6d4161a595d779a9fc96cc in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.273048 25727 raft_consensus.cc:351] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.273289 25727 raft_consensus.cc:628] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:54.273756 25727 consensus_queue.cc:227] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.278126 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.282117 25470 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.283782 25727 ts_tablet_manager.cc:1073] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I0619 00:11:54.284162 25727 ts_tablet_manager.cc:1028] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:54.284538 25727 tablet_bootstrap.cc:427] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:54.289830 25885 raft_consensus.cc:457] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.290087 25885 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.965s (starting election)
I0619 00:11:54.290330 25885 raft_consensus.cc:479] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.291288 25598 ts_tablet_manager.cc:1173] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.291345 25885 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.291527 25914 raft_consensus.cc:457] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.291779 25914 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.940s (starting election)
I0619 00:11:54.291918 25885 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.291895 25796 raft_consensus.cc:457] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.292138 25796 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.995s (starting election)
I0619 00:11:54.292017 25914 raft_consensus.cc:479] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.292379 25796 raft_consensus.cc:479] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.293049 25914 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.292913 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.293395 25796 leader_election.cc:243] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.293581 25736 raft_consensus.cc:2069] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.294035 25914 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.294066 25796 leader_election.cc:243] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.294507 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.295086 25487 raft_consensus.cc:2206] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:54.295850 25084 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:54.296150 25290 leader_election.cc:393] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:54.296497 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:54.297452 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.297451 25085 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:54.298167 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:54.298338 25736 ts_tablet_manager.cc:1369] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.300209 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_1074470b12de48baa6a1264ad788d237" }
I0619 00:11:54.302784 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "30a9d9fda8d34caeb8242c372439321b" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:54.303314 25622 raft_consensus.cc:2206] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:54.301414 25336 ts_tablet_manager.cc:1173] T 756057a7c34a4c61a261deaf849e433a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.304386 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.305416 25084 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
I0619 00:11:54.307301 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC with a delay of 527 ms (attempt = 6)
I0619 00:11:54.308665 25325 ts_tablet_manager.cc:1173] T d0624684b6b04ba3be260b2705dc00e1 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.308989 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.309020 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775f0b6b309c49278066a5e7cf623af2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.312091 25914 raft_consensus.cc:457] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:54.307636 25071 catalog_manager.cc:2910] Async tablet task fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fc7686daae274e0dba87703fe6924805 consensus state has no leader
W0619 00:11:54.313375 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.312332 25914 raft_consensus.cc:2732] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.180s (starting election)
I0619 00:11:54.313735 25914 raft_consensus.cc:479] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.314748 25914 leader_election.cc:243] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.315104 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 109 ms (attempt = 3)
W0619 00:11:54.315394 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.315666 25914 leader_election.cc:243] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.315831 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 522 ms (attempt = 6)
W0619 00:11:54.316081 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c consensus state has no leader
W0619 00:11:54.317020 25553 leader_election.cc:306] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.263945 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(0474be659e5e4a4cbfc181b8c812b1eb)
W0619 00:11:54.317714 25290 leader_election.cc:306] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:54.317940 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.318171 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.318425 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.318413 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet fc7686daae274e0dba87703fe6924805
I0619 00:11:54.318702 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.318938 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.319517 25730 raft_consensus.cc:2042] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.319504 25290 leader_election.cc:393] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:54.319761 25290 leader_election.cc:270] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.319854 25730 raft_consensus.cc:2069] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.320482 25914 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.930s (election complete)
I0619 00:11:54.320719 25914 raft_consensus.cc:2784] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.320951 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775f0b6b309c49278066a5e7cf623af2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:54.321398 25623 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.355s (vote granted)
I0619 00:11:54.321607 25623 raft_consensus.cc:2245] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
W0619 00:11:54.318943 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 4a88e394b199431e8b794a1b1aec847f because the tablet was not found. No further retry: Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:54.322458 25730 ts_tablet_manager.cc:1369] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.323691 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) successfully deleted
I0619 00:11:54.299283 25899 tablet.cc:1200] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:54.303952 25725 tablet_bootstrap.cc:427] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
W0619 00:11:54.310346 25419 leader_election.cc:306] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:54.324314 25290 leader_election.cc:306] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:54.324331 25725 ts_tablet_manager.cc:1045] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.143s	user 0.005s	sys 0.001s
I0619 00:11:54.324573 25290 leader_election.cc:270] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.324760 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ad3a5796593e44cc8806d94c7b09239a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:54.325754 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ad3a5796593e44cc8806d94c7b09239a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.326290 25086 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:54.326071 25725 raft_consensus.cc:325] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.326369 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_30bcafcc4a6f46e99005fd47447e7dbe"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:54.326813 25725 raft_consensus.cc:351] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.326964 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:54.327080 25725 raft_consensus.cc:628] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:54.327225 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_30bcafcc4a6f46e99005fd47447e7dbe in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.327783 25725 consensus_queue.cc:227] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.321897 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.331516 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.331706 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.333173 25995 raft_consensus.cc:2732] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.907s (election complete)
I0619 00:11:54.333482 25995 raft_consensus.cc:2487] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.333861 25727 tablet_bootstrap.cc:578] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
W0619 00:11:54.339035 25558 leader_election.cc:306] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.339663 25558 leader_election.cc:270] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.340303 25725 ts_tablet_manager.cc:1073] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.001s
I0619 00:11:54.340479 25885 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.889s (election complete)
I0619 00:11:54.340672 25725 ts_tablet_manager.cc:1028] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:54.340682 25914 raft_consensus.cc:2487] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.340713 25885 raft_consensus.cc:2487] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.341014 25725 tablet_bootstrap.cc:427] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:54.341300 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 03522c48c50943b38cae5aa88562e583 because the tablet was not found. No further retry: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:54.345566 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.345726 25724 tablet_bootstrap.cc:578] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:54.346482 25741 ts_tablet_manager.cc:1382] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I0619 00:11:54.346989 25741 log.cc:986] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/36f5eec972454080b7b434b4df7e62e3
I0619 00:11:54.350335 25602 ts_tablet_manager.cc:1173] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:54.351325 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.341737 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet d6c3c7f783cb43d59b46502261aeced3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet d6c3c7f783cb43d59b46502261aeced3 already in progress: creating tablet
I0619 00:11:54.352370 25071 catalog_manager.cc:2891] Scheduling retry of d6c3c7f783cb43d59b46502261aeced3 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 31 ms (attempt = 1)
I0619 00:11:54.356197 25598 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.356079 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.357790 25467 ts_tablet_manager.cc:1173] T cbd67634f5704165b5bb948501697f0c P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.359634 25923 tablet.cc:1200] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:54.358227 25067 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 160 ms (attempt = 4)
W0619 00:11:54.363829 25067 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:54.365306 25940 raft_consensus.cc:457] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.365571 25940 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.831s (starting election)
I0619 00:11:54.365847 25940 raft_consensus.cc:479] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.366843 25940 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.367216 25940 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:54.367780 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.369035 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.368968 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7f60ea5c5def47ee846d0186bbc90c18" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.369833 25487 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 3.129s (vote granted)
I0619 00:11:54.369817 25325 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.370043 25487 raft_consensus.cc:2245] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:54.370412 25725 tablet_bootstrap.cc:578] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:54.371544 25290 leader_election.cc:379] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.371819 25290 leader_election.cc:270] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:54.372328 25995 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.790s (election complete)
I0619 00:11:54.372575 25995 raft_consensus.cc:2539] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:54.372829 25995 raft_consensus.cc:457] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:54.373075 25995 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.671s (starting election)
I0619 00:11:54.373296 25995 raft_consensus.cc:2784] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.373014 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:54.377439 25419 consensus_peers.cc:434] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 7c57c5ca55f24621b66197a3698504bd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:54.378895 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
W0619 00:11:54.379058 25290 leader_election.cc:306] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:54.379462 25083 catalog_manager.cc:3663] Got report from deleted tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:54.379616 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:54.379920 25290 consensus_peers.cc:434] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet cb57d196f0724247b6a7f9a34be8377a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:54.380015 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.380007 25741 ts_tablet_manager.cc:1403] T 36f5eec972454080b7b434b4df7e62e3 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:54.380147 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb57d196f0724247b6a7f9a34be8377a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:54.381750 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 232) took 1813ms. Request Metrics: {"tablet-delete.queue_time_us":1605893}
I0619 00:11:54.383438 25741 tablet_replica.cc:261] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:54.383029 25727 tablet_bootstrap.cc:427] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:54.384059 25727 ts_tablet_manager.cc:1045] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.100s	user 0.005s	sys 0.001s
I0619 00:11:54.385123 25325 tablet_service.cc:800] Processing DeleteTablet for tablet cb57d196f0724247b6a7f9a34be8377a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:54.385349 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.385535 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 36f5eec972454080b7b434b4df7e62e3 (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) successfully deleted
I0619 00:11:54.385460 25727 raft_consensus.cc:325] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.386096 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:54.386217 25727 raft_consensus.cc:351] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.386456 25727 raft_consensus.cc:628] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:54.387428 25727 consensus_queue.cc:227] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.390394 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.393332 25558 leader_election.cc:306] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.393836 25558 leader_election.cc:270] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.394266 25885 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.931s (election complete)
I0619 00:11:54.394486 25885 raft_consensus.cc:2487] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.395519 25084 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a] alter complete (version 1)
I0619 00:11:54.395848 25736 ts_tablet_manager.cc:1382] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:54.396088 25736 log.cc:986] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/4a88e394b199431e8b794a1b1aec847f
I0619 00:11:54.396786 25736 ts_tablet_manager.cc:1403] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:54.397640 25724 tablet_bootstrap.cc:427] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:54.398013 25724 ts_tablet_manager.cc:1045] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.139s	user 0.004s	sys 0.001s
I0619 00:11:54.399161 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 1073 ms (attempt = 7)
W0619 00:11:54.399420 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:54.399636 25724 raft_consensus.cc:325] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.400460 25724 raft_consensus.cc:351] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.400692 25602 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.400707 25724 raft_consensus.cc:628] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:54.405238 25724 consensus_queue.cc:227] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.407296 25724 ts_tablet_manager.cc:1073] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
W0619 00:11:54.407730 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.408227 25724 ts_tablet_manager.cc:1028] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:54.408594 25724 tablet_bootstrap.cc:427] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:54.409837 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.411859 25724 tablet_bootstrap.cc:578] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:54.414808 25730 ts_tablet_manager.cc:1382] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:54.415086 25730 log.cc:986] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:54.418012 25085 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 1 got 0
I0619 00:11:54.418103 25071 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 263 ms (attempt = 5)
W0619 00:11:54.418412 25071 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:54.418844 25085 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 33 ms (attempt = 1)
W0619 00:11:54.419118 25085 catalog_manager.cc:3945] Failed to send 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.417088 25602 tablet_service.cc:759] Processing CreateTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 (table=default.table_3e0e0119254a4be881b82ecf36174935 [id=a4ae0343c119473d90e61fab1e6df662]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.419705 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b82f9befb9144a7b4be080ac6d92a73. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.402504 25725 tablet_bootstrap.cc:427] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:54.420557 25725 ts_tablet_manager.cc:1045] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.080s	user 0.005s	sys 0.000s
W0619 00:11:54.403707 25423 leader_election.cc:306] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:54.421149 25423 leader_election.cc:270] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.421515 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 205) took 2134ms. Request Metrics: {"tablet-delete.queue_time_us":1967645}
W0619 00:11:54.421851 25423 raft_consensus.cc:2442] T 4a88e394b199431e8b794a1b1aec847f P 10bda0b4856f4441b5aed24cdc4764a8: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:11:54.421957 25725 raft_consensus.cc:325] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.422871 25725 raft_consensus.cc:351] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.423105 25725 raft_consensus.cc:628] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:54.422231 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 202) took 2136ms. Request Metrics: {"tablet-delete.queue_time_us":2134553}
I0619 00:11:54.423571 25725 consensus_queue.cc:227] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.424499 25467 tablet_service.cc:759] Processing CreateTablet for tablet ef1566aa89b24620af70f2322df0dd56 (table=default.table_f05cbe4e96c04214a2f1a82f57ee9b77 [id=0c557a51d7e24b90ac96183eda7f4785]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.426264 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ef1566aa89b24620af70f2322df0dd56. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.427640 25725 ts_tablet_manager.cc:1073] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I0619 00:11:54.428014 25725 ts_tablet_manager.cc:1028] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:54.428341 25725 tablet_bootstrap.cc:427] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:54.428725 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 136 ms (attempt = 4)
W0619 00:11:54.428989 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.424450 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 199) took 2140ms. Request Metrics: {"tablet-delete.queue_time_us":2126649}
I0619 00:11:54.415582 25730 ts_tablet_manager.cc:1403] T 0474be659e5e4a4cbfc181b8c812b1eb P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:54.388181 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:54.378309 25995 raft_consensus.cc:479] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.416538 25325 tablet_service.cc:759] Processing CreateTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 (table=default.table_3e0e0119254a4be881b82ecf36174935 [id=a4ae0343c119473d90e61fab1e6df662]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.421306 25336 tablet_service.cc:759] Processing CreateTablet for tablet 14179adec4b241be87f42625f5edefc6 (table=default.table_d036ca047e034c39958950ab500d6004 [id=6f8996225b924c6db7ebdeaebb35340e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.432363 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:54.432858 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b82f9befb9144a7b4be080ac6d92a73. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.432894 25995 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.433507 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14179adec4b241be87f42625f5edefc6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.433501 25995 leader_election.cc:243] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.435077 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_559173a19fea4dfea574c9469397df05" } new_table_name: "default.table_047cd2779baa44c18f0d11f4e4ae83a2" alter_external_catalogs: true
I0619 00:11:54.434561 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7f60ea5c5def47ee846d0186bbc90c18" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:54.435874 25487 raft_consensus.cc:2784] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.436117 25487 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 3.368s (vote granted)
I0619 00:11:54.440656 25487 raft_consensus.cc:2245] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:54.440778 25083 catalog_manager.cc:3663] Got report from deleted tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:54.441442 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:54.441401 25598 tablet_service.cc:759] Processing CreateTablet for tablet 14179adec4b241be87f42625f5edefc6 (table=default.table_d036ca047e034c39958950ab500d6004 [id=6f8996225b924c6db7ebdeaebb35340e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.441989 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14179adec4b241be87f42625f5edefc6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.442190 25290 leader_election.cc:379] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.442414 25290 leader_election.cc:270] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:54.443480 25331 tablet_service.cc:759] Processing CreateTablet for tablet 28f9fefa25d14aa089c34c9a0825946e (table=default.table_b2764b59c43146a8ae54b361c93b3406 [id=464afb6b12f4446b89a9682f1d407e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.444108 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 28f9fefa25d14aa089c34c9a0825946e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.445385 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:54.449486 25085 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 17 ms (attempt = 1)
W0619 00:11:54.450021 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:54.450366 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:54.450631 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:54.450832 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:54.451014 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(cb57d196f0724247b6a7f9a34be8377a)
I0619 00:11:54.451186 25741 raft_consensus.cc:2042] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:54.451386 25741 consensus_peers.cc:456] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:54.451450 25600 tablet_service.cc:759] Processing CreateTablet for tablet 28f9fefa25d14aa089c34c9a0825946e (table=default.table_b2764b59c43146a8ae54b361c93b3406 [id=464afb6b12f4446b89a9682f1d407e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.451622 25741 consensus_peers.cc:456] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
I0619 00:11:54.451869 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_eb0f587538da43c39a0017ca0930cda9" }
I0619 00:11:54.453872 25084 catalog_manager.cc:3663] Got report from deleted tablet a284677c4b7f46e18e25edee6024e3eb (table default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:54.461649 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 28f9fefa25d14aa089c34c9a0825946e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.455018 25067 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 42 ms (attempt = 2)
W0619 00:11:54.462594 25067 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.459167 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_27e7515742a84639bd10967f26b8d93f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:54.463389 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_27e7515742a84639bd10967f26b8d93f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.463459 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 168) took 1498ms. Request Metrics: {"tablet-delete.queue_time_us":1297908}
I0619 00:11:54.454221 25331 ts_tablet_manager.cc:1173] T 28f9fefa25d14aa089c34c9a0825946e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.464370 25325 ts_tablet_manager.cc:1173] T 2b82f9befb9144a7b4be080ac6d92a73 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.464987 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:54.467483 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 45 ms (attempt = 2)
W0619 00:11:54.467777 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:54.468264 25741 pending_rounds.cc:63] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Trying to abort 1 pending transactions.
I0619 00:11:54.468457 25741 pending_rounds.cc:68] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390912982757376 op_type: NO_OP noop_request { }
I0619 00:11:54.461225 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 207) took 1995ms. Request Metrics: {"tablet-delete.queue_time_us":1991990}
I0619 00:11:54.469202 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 209) took 1979ms. Request Metrics: {"tablet-delete.queue_time_us":1977787}
I0619 00:11:54.469370 25336 ts_tablet_manager.cc:1173] T 14179adec4b241be87f42625f5edefc6 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.473017 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 211) took 1977ms. Request Metrics: {"tablet-delete.queue_time_us":1972491}
I0619 00:11:54.474481 25730 tablet_replica.cc:261] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:54.476141 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 215) took 1893ms. Request Metrics: {"tablet-delete.queue_time_us":1892647}
I0619 00:11:54.476658 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:54.468782 25741 raft_consensus.cc:2622] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:54.478776 25741 raft_consensus.cc:2069] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.479326 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:54.479750 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:54.480021 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet b34043f14f1c4edd97a1d4689ced5b32
I0619 00:11:54.480389 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.480801 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.479974 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) successfully deleted
I0619 00:11:54.481274 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:54.481526 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:54.481170 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.481757 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:54.481950 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:54.470170 25724 tablet_bootstrap.cc:427] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:54.482429 25724 ts_tablet_manager.cc:1045] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.074s	user 0.003s	sys 0.002s
I0619 00:11:54.430588 25725 tablet_bootstrap.cc:578] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:54.483958 25724 raft_consensus.cc:325] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.484845 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.486029 25724 raft_consensus.cc:351] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.486366 25724 raft_consensus.cc:628] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:54.487550 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.475378 25600 ts_tablet_manager.cc:1173] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.488209 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) successfully deleted
I0619 00:11:54.506822 25598 ts_tablet_manager.cc:1173] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.507256 25741 ts_tablet_manager.cc:1369] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.489338 25604 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.489015 25599 tablet_service.cc:759] Processing CreateTablet for tablet dba1210d9c22454f8c9e0b0293583f0c (table=default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.488221 25601 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.485205 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 216) took 1901ms. Request Metrics: {"tablet-delete.queue_time_us":1896461}
I0619 00:11:54.508083 25470 tablet_service.cc:759] Processing CreateTablet for tablet 31b36efa3dbd4aa983b3a396aa9d62f1 (table=default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.508293 25730 raft_consensus.cc:2042] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.508639 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dba1210d9c22454f8c9e0b0293583f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.508821 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 31b36efa3dbd4aa983b3a396aa9d62f1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.508672 25730 raft_consensus.cc:2069] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.509101 25736 tablet_replica.cc:261] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:54.509582 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.509974 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.510285 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.510599 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.510996 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.511323 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.511442 25467 ts_tablet_manager.cc:1173] T ef1566aa89b24620af70f2322df0dd56 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.511751 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:54.512066 25736 raft_consensus.cc:2042] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.512496 25736 raft_consensus.cc:2069] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.487278 25724 consensus_queue.cc:227] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.516019 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.518548 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.521890 25602 ts_tablet_manager.cc:1173] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.524304 25724 ts_tablet_manager.cc:1073] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.000s
I0619 00:11:54.525754 25470 ts_tablet_manager.cc:1173] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.523011 25727 ts_tablet_manager.cc:1073] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.139s	user 0.005s	sys 0.004s
I0619 00:11:54.507014 25067 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 96 ms (attempt = 3)
I0619 00:11:54.513684 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 102 ms (attempt = 3)
I0619 00:11:54.518069 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]). Expected version 1 got 0
W0619 00:11:54.530802 25067 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
W0619 00:11:54.530861 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:54.531235 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC with a delay of 539 ms (attempt = 6)
W0619 00:11:54.531484 25067 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:54.531857 25067 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 302 ms (attempt = 5)
I0619 00:11:54.532024 25940 raft_consensus.cc:2732] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.770s (election complete)
W0619 00:11:54.532117 25067 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:54.531538 25085 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 30 ms (attempt = 1)
W0619 00:11:54.532378 25085 catalog_manager.cc:3945] Failed to send 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.533007 25947 raft_consensus.cc:457] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.525198 25623 raft_consensus.cc:1081] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:54.533265 25947 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.541s (starting election)
I0619 00:11:54.533535 25947 raft_consensus.cc:479] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.534515 25947 leader_election.cc:243] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.534927 25947 leader_election.cc:243] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.535830 25724 ts_tablet_manager.cc:1028] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:54.536178 25724 tablet_bootstrap.cc:427] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:54.537071 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]). Expected version 1 got 0
I0619 00:11:54.537634 25725 tablet_bootstrap.cc:427] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:54.537783 25085 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 24 ms (attempt = 1)
I0619 00:11:54.538060 25725 ts_tablet_manager.cc:1045] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.110s	user 0.005s	sys 0.001s
W0619 00:11:54.538036 25085 catalog_manager.cc:3945] Failed to send 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
W0619 00:11:54.538463 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.528537 25470 tablet_service.cc:759] Processing CreateTablet for tablet b8bc8d4a970843258ab5e9530f425d0f (table=default.table_b4f172b08ffd4adb9f52e785f33e3076 [id=eca38c9105fd4a3a8d9b1e260ebc045c]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:54.539559 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 97386.9us
I0619 00:11:54.539814 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b8bc8d4a970843258ab5e9530f425d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.539492 25725 raft_consensus.cc:325] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.540148 25725 raft_consensus.cc:351] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.540344 25725 raft_consensus.cc:628] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:54.540879 25725 consensus_queue.cc:227] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.545336 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_34ae54d8d2fc4703a6fd963ddd934b4d" } new_table_name: "default.table_1139cb4bdb734efd950a2c4f0a1e4b95" alter_external_catalogs: true
I0619 00:11:54.546416 25467 tablet_service.cc:800] Processing DeleteTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.550858 25727 ts_tablet_manager.cc:1028] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:54.551316 25727 tablet_bootstrap.cc:427] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:54.553635 25290 leader_election.cc:306] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:54.554735 25336 tablet_service.cc:759] Processing CreateTablet for tablet dba1210d9c22454f8c9e0b0293583f0c (table=default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.555320 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dba1210d9c22454f8c9e0b0293583f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.556632 25083 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 20 ms (attempt = 1)
W0619 00:11:54.556917 25083 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:54.558372 25084 catalog_manager.cc:3663] Got report from deleted tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
W0619 00:11:54.558549 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times.
I0619 00:11:54.558652 25085 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
W0619 00:11:54.554376 25423 consensus_peers.cc:434] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 36f5eec972454080b7b434b4df7e62e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36f5eec972454080b7b434b4df7e62e3. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:54.559383 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:54.559540 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:54.560274 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_54575c07db144677bb4aeb1144a588b5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:54.561260 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_54575c07db144677bb4aeb1144a588b5 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.561518 25736 ts_tablet_manager.cc:1369] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.562603 25069 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 39 ms (attempt = 2)
W0619 00:11:54.562939 25069 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.563701 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 41 ms (attempt = 2)
W0619 00:11:54.563984 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.527217 25730 ts_tablet_manager.cc:1369] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.556542 25885 raft_consensus.cc:457] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.564890 25885 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.711s (starting election)
I0619 00:11:54.565141 25885 raft_consensus.cc:479] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.566108 25885 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.566483 25885 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.560721 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a33940e5347489993ab4679e9aeafde" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:54.566977 25467 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.567071 25623 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.167s (vote granted)
I0619 00:11:54.567291 25623 raft_consensus.cc:2245] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:54.567195 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a33940e5347489993ab4679e9aeafde" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.560088 25601 tablet_service.cc:759] Processing CreateTablet for tablet 8b437c976cfd41aa8f2f2f38b3762005 (table=default.table_e874b4b421a446c48acc51dc98a14038 [id=48e78cf9fa7840a3b2b84621f7c06cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.568558 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8b437c976cfd41aa8f2f2f38b3762005. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:54.568583 25419 leader_election.cc:306] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:54.571316 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.571956 25725 ts_tablet_manager.cc:1073] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.034s	user 0.003s	sys 0.001s
I0619 00:11:54.572391 25725 ts_tablet_manager.cc:1028] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:54.572341 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 283 ms (attempt = 5)
W0619 00:11:54.572722 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.574431 25325 tablet_service.cc:759] Processing CreateTablet for tablet fa0ab4b6eef04ae08eb8994b49ca531e (table=default.table_6ebe326fe1fe4f3580d3f06acc67821f [id=b9bfdc8e55474269827835656bfecf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.575063 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fa0ab4b6eef04ae08eb8994b49ca531e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.572836 25725 tablet_bootstrap.cc:427] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:54.575892 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.577554 25725 tablet_bootstrap.cc:578] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:54.579239 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 77 ms (attempt = 2)
W0619 00:11:54.579509 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:54.532258 25940 raft_consensus.cc:2539] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:54.582605 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:54.583458 25084 master_service.cc:523] ReplaceTablet: received request to replace tablet 86959e7dda374ba9ac23d3c1fd30d993 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:54.584143 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:54.584805 25071 catalog_manager.cc:2891] Scheduling retry of f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC with a delay of 4097 ms (attempt = 9)
W0619 00:11:54.585062 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
W0619 00:11:54.585531 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
W0619 00:11:54.586203 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet c18355df78d344839a56189c7549ab1f because the tablet was not found. No further retry: Not found: Tablet not found: c18355df78d344839a56189c7549ab1f
I0619 00:11:54.586401 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_58781b6793a04331a5a69cdbb7cd2cc4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:54.586910 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 35acf55f35064c1491122a13ff9f5744 because the tablet was not found. No further retry: Not found: Tablet not found: 35acf55f35064c1491122a13ff9f5744
W0619 00:11:54.587312 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_58781b6793a04331a5a69cdbb7cd2cc4 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.587133 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d9245875d6214b88a63997a3d300f512"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:54.587997 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d9245875d6214b88a63997a3d300f512 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.591128 25724 tablet_bootstrap.cc:578] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:54.592144 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54adb588955b45b79de9752f254728ea" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.594053 25423 leader_election.cc:379] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.596153 25423 leader_election.cc:270] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:54.598137 25847 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.228s (election complete)
I0619 00:11:54.598371 25847 raft_consensus.cc:2539] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:54.598577 25847 raft_consensus.cc:457] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:54.598798 25847 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.044s (starting election)
I0619 00:11:54.598989 25847 raft_consensus.cc:2784] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.603032 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3e0e0119254a4be881b82ecf36174935" }
I0619 00:11:54.603826 25069 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 77 ms (attempt = 3)
W0619 00:11:54.604097 25069 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.605031 25847 raft_consensus.cc:479] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.605969 25847 leader_election.cc:243] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.607847 25847 leader_election.cc:243] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.608145 25083 catalog_manager.cc:3663] Got report from deleted tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c): Sending delete request for this tablet
I0619 00:11:54.608129 25601 ts_tablet_manager.cc:1173] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.608808 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
I0619 00:11:54.609691 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 81 ms (attempt = 3)
W0619 00:11:54.609972 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
W0619 00:11:54.587324 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 981d862d1bab4b14bb10e6fc2174e1cc already in progress: creating tablet
I0619 00:11:54.612010 25071 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 34 ms (attempt = 1)
I0619 00:11:54.612522 25467 tablet_service.cc:759] Processing CreateTablet for tablet d0624684b6b04ba3be260b2705dc00e1 (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:54.612799 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 54adb588955b45b79de9752f254728ea with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 54adb588955b45b79de9752f254728ea already in progress: creating tablet
I0619 00:11:54.613035 25071 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 26 ms (attempt = 1)
I0619 00:11:54.613176 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d0624684b6b04ba3be260b2705dc00e1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.613659 25602 tablet_service.cc:759] Processing CreateTablet for tablet fa0ab4b6eef04ae08eb8994b49ca531e (table=default.table_6ebe326fe1fe4f3580d3f06acc67821f [id=b9bfdc8e55474269827835656bfecf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.614059 25724 tablet_bootstrap.cc:427] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:54.614445 25724 ts_tablet_manager.cc:1045] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.078s	user 0.005s	sys 0.000s
I0619 00:11:54.615984 25724 raft_consensus.cc:325] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.616750 25724 raft_consensus.cc:351] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.616958 25724 raft_consensus.cc:628] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:54.618445 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.618834 25725 tablet_bootstrap.cc:427] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:54.618227 25724 consensus_queue.cc:227] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.619206 25725 ts_tablet_manager.cc:1045] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.047s	user 0.004s	sys 0.000s
I0619 00:11:54.620558 25725 raft_consensus.cc:325] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.621222 25725 raft_consensus.cc:351] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.622620 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54adb588955b45b79de9752f254728ea" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.622892 25468 tablet_service.cc:759] Processing CreateTablet for tablet 756057a7c34a4c61a261deaf849e433a (table=default.table_252d88d7d98b495e80affd5dc6eeb739 [id=f8e9404177f44b2cba84c5146b780133]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.623445 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 756057a7c34a4c61a261deaf849e433a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.627279 25331 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.627799 25067 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 157 ms (attempt = 4)
W0619 00:11:54.628075 25067 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
W0619 00:11:54.631832 25558 leader_election.cc:306] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:54.632457 25553 leader_election.cc:306] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.632730 25553 leader_election.cc:270] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.633481 25885 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.919s (election complete)
I0619 00:11:54.633730 25885 raft_consensus.cc:2487] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.633816 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 163 ms (attempt = 4)
W0619 00:11:54.634076 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
W0619 00:11:54.634075 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.638685 25419 consensus_peers.cc:434] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 36f5eec972454080b7b434b4df7e62e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36f5eec972454080b7b434b4df7e62e3. Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:11:54.639390 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.639936 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.640866 25331 tablet_service.cc:759] Processing CreateTablet for tablet 8b437c976cfd41aa8f2f2f38b3762005 (table=default.table_e874b4b421a446c48acc51dc98a14038 [id=48e78cf9fa7840a3b2b84621f7c06cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.641451 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8b437c976cfd41aa8f2f2f38b3762005. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.641523 25083 catalog_manager.cc:3663] Got report from deleted tablet 54373917567a479d93dc8a3fd006d2fe (table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:54.642278 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:54.646936 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775f0b6b309c49278066a5e7cf623af2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:54.647400 25624 raft_consensus.cc:2784] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.647667 25624 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.461s (vote granted)
I0619 00:11:54.648671 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:54.650413 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.652567 25624 raft_consensus.cc:2245] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:54.636582 25470 ts_tablet_manager.cc:1173] T b8bc8d4a970843258ab5e9530f425d0f P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.614451 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fa0ab4b6eef04ae08eb8994b49ca531e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.654263 25336 ts_tablet_manager.cc:1173] T dba1210d9c22454f8c9e0b0293583f0c P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.655087 25468 ts_tablet_manager.cc:1173] T 756057a7c34a4c61a261deaf849e433a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.658541 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.658953 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 106 ms (attempt = 3)
W0619 00:11:54.659211 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:54.659422 25730 ts_tablet_manager.cc:1382] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:54.659672 25730 log.cc:986] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/d62715e230154e2abe126ef97feec7ce
I0619 00:11:54.660189 25730 ts_tablet_manager.cc:1403] T d62715e230154e2abe126ef97feec7ce P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:54.661962 25325 ts_tablet_manager.cc:1173] T fa0ab4b6eef04ae08eb8994b49ca531e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.662766 25331 ts_tablet_manager.cc:1173] T 8b437c976cfd41aa8f2f2f38b3762005 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.663269 25467 ts_tablet_manager.cc:1173] T d0624684b6b04ba3be260b2705dc00e1 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.663280 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:54.663730 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:54.665345 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 981d862d1bab4b14bb10e6fc2174e1cc from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:54.666071 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_6bb13c214301415189d296eaabe6884b" } new_table_name: "default.table_c57b5014fe19429b8fde87e3fee4d2b7" alter_external_catalogs: true
I0619 00:11:54.666731 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.669178 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 86959e7dda374ba9ac23d3c1fd30d993
I0619 00:11:54.669456 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:54.669837 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:54.670181 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:54.670472 25084 catalog_manager.cc:4596] ReplaceTablet: tablet 86959e7dda374ba9ac23d3c1fd30d993 deleted and replaced by tablet bae6d308dae54b0394a364d2988ec6b3
I0619 00:11:54.672096 25083 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:54.673055 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 2b82f9befb9144a7b4be080ac6d92a73
I0619 00:11:54.673465 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.673880 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.674522 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.668421 25847 consensus_queue.cc:963] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:11:54.677887 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.678866 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.678889 25724 ts_tablet_manager.cc:1073] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.064s	user 0.004s	sys 0.000s
I0619 00:11:54.679333 25724 ts_tablet_manager.cc:1028] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:54.621438 25725 raft_consensus.cc:628] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:54.679527 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:54.679761 25724 tablet_bootstrap.cc:427] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:54.679869 25725 consensus_queue.cc:227] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.681604 25725 ts_tablet_manager.cc:1073] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.062s	user 0.004s	sys 0.000s
I0619 00:11:54.681998 25725 ts_tablet_manager.cc:1028] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:54.661607 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 169) took 1695ms. Request Metrics: {"tablet-delete.queue_time_us":1507898}
I0619 00:11:54.683265 25725 tablet_bootstrap.cc:427] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:54.684157 25899 tablet.cc:1195] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
I0619 00:11:54.683959 25730 tablet_replica.cc:261] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:54.685734 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:54.686583 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:54.681798 25069 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 159 ms (attempt = 4)
I0619 00:11:54.686928 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(7c57c5ca55f24621b66197a3698504bd)
W0619 00:11:54.686928 25069 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.683642 25085 catalog_manager.cc:3663] Got report from deleted tablet 54373917567a479d93dc8a3fd006d2fe (table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:54.687130 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:54.687433 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:54.687675 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:54.689373 25725 tablet_bootstrap.cc:578] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:54.691548 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 171 ms (attempt = 4)
W0619 00:11:54.691855 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.693414 25467 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.694844 25599 ts_tablet_manager.cc:1173] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.696132 25071 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 524 ms (attempt = 6)
W0619 00:11:54.696286 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:54.696382 25071 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:54.696851 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) successfully deleted
I0619 00:11:54.697176 25847 raft_consensus.cc:457] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.697420 25847 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.146s (starting election)
I0619 00:11:54.697266 25083 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:54.697867 25724 tablet_bootstrap.cc:578] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:54.697685 25847 raft_consensus.cc:479] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.698422 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:54.698657 25847 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.699043 25847 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.699996 25598 tablet_service.cc:759] Processing CreateTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 (table=default.table_30bcafcc4a6f46e99005fd47447e7dbe [id=6ba1ca844e8d4a9ab4157314c5e4f543]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.700143 25736 ts_tablet_manager.cc:1382] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:54.700264 25604 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.700379 25736 log.cc:986] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:54.700582 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3e7c4b61f5314e40ad71371155a50b05. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.700856 25736 ts_tablet_manager.cc:1403] T 0474be659e5e4a4cbfc181b8c812b1eb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:54.700919 25600 tablet_service.cc:759] Processing CreateTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 (table=default.table_53e73552aa6d4161a595d779a9fc96cc [id=1697152e5d9f409ab68219b93b01250f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.701464 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8ca3a0771b9e41eeb0d14d886f970159. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.702255 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 217) took 2117ms. Request Metrics: {"tablet-delete.queue_time_us":1909199}
I0619 00:11:54.702831 25736 tablet_replica.cc:261] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:54.702772 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.703054 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.703279 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.653290 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775f0b6b309c49278066a5e7cf623af2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:54.664000 25940 raft_consensus.cc:593] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:54.704016 25940 consensus_queue.cc:204] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.705407 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.665168 26008 raft_consensus.cc:457] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.686465 25741 ts_tablet_manager.cc:1382] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:54.705978 26008 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.813s (starting election)
I0619 00:11:54.692986 26022 tablet.cc:1200] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:54.668269 25727 tablet_bootstrap.cc:578] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:54.706357 26008 raft_consensus.cc:479] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.706064 25741 log.cc:986] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:54.707352 26008 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.707757 26008 leader_election.cc:243] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:54.664343 25290 consensus_peers.cc:434] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet 03522c48c50943b38cae5aa88562e583. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:54.708173 25741 ts_tablet_manager.cc:1403] T cb57d196f0724247b6a7f9a34be8377a P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:54.708365 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 44093.1us
W0619 00:11:54.709098 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.710150 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.688535 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:54.712798 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.713598 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:54.713826 25730 raft_consensus.cc:2042] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:54.714192 25730 raft_consensus.cc:2069] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:54.714736 25290 consensus_peers.cc:434] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 03522c48c50943b38cae5aa88562e583. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583. Retrying in the next heartbeat period. Already tried 2 times.
W0619 00:11:54.715427 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.715862 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:54.717048 25330 tablet_service.cc:759] Processing CreateTablet for tablet a858d8f5709648a3ba7649315ccc9662 (table=default.table_27e7515742a84639bd10967f26b8d93f [id=2f4c8afb95b945138a65de871be4b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.717280 25336 tablet_service.cc:759] Processing CreateTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 (table=default.table_53e73552aa6d4161a595d779a9fc96cc [id=1697152e5d9f409ab68219b93b01250f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.717638 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a858d8f5709648a3ba7649315ccc9662. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.718142 25325 tablet_service.cc:759] Processing CreateTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 (table=default.table_30bcafcc4a6f46e99005fd47447e7dbe [id=6ba1ca844e8d4a9ab4157314c5e4f543]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.718524 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.718879 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3e7c4b61f5314e40ad71371155a50b05. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.721302 25885 raft_consensus.cc:457] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.721525 25885 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.615s (starting election)
I0619 00:11:54.721853 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8ca3a0771b9e41eeb0d14d886f970159. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.721786 25885 raft_consensus.cc:479] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.722162 25602 ts_tablet_manager.cc:1173] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.722789 25885 leader_election.cc:243] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.723204 25885 leader_election.cc:243] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.725054 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19ad911ec6704aef85d9b7f172f08bdc" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:54.726284 25553 leader_election.cc:306] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:54.704989 25419 leader_election.cc:306] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:54.730146 25419 leader_election.cc:306] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.714186 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7c57c5ca55f24621b66197a3698504bd" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.731122 25489 raft_consensus.cc:2206] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:54.732936 25730 ts_tablet_manager.cc:1369] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.733248 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.733501 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.733856 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.734100 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.734380 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(36f5eec972454080b7b434b4df7e62e3)
I0619 00:11:54.734630 25736 raft_consensus.cc:2042] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:54.734892 25736 consensus_peers.cc:456] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:54.735136 25736 consensus_peers.cc:456] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:54.735671 25330 ts_tablet_manager.cc:1173] T a858d8f5709648a3ba7649315ccc9662 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.735685 25736 raft_consensus.cc:2069] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.739451 25599 tablet_service.cc:759] Processing CreateTablet for tablet a858d8f5709648a3ba7649315ccc9662 (table=default.table_27e7515742a84639bd10967f26b8d93f [id=2f4c8afb95b945138a65de871be4b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.740063 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a858d8f5709648a3ba7649315ccc9662. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.740610 25083 catalog_manager.cc:3663] Got report from deleted tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:54.740720 25601 tablet_service.cc:759] Processing CreateTablet for tablet 9704177a036c469ab3d14211271cfd84 (table=default.table_ad3a5796593e44cc8806d94c7b09239a [id=d550ebb246c14e28b1093f5ac04a6cea]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.741077 25083 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]). Expected version 1 got 0
I0619 00:11:54.741300 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9704177a036c469ab3d14211271cfd84. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.741961 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:54.745014 25290 leader_election.cc:306] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.746723 25290 leader_election.cc:393] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:54.747016 25290 leader_election.cc:270] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.747484 26008 raft_consensus.cc:2732] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.894s (election complete)
I0619 00:11:54.747732 26008 raft_consensus.cc:2784] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.750743 25724 tablet_bootstrap.cc:427] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:54.751251 25724 ts_tablet_manager.cc:1045] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.072s	user 0.004s	sys 0.002s
I0619 00:11:54.753476 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.752899 26008 raft_consensus.cc:2487] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.752890 25336 ts_tablet_manager.cc:1173] T 8ca3a0771b9e41eeb0d14d886f970159 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.754060 25330 tablet_service.cc:759] Processing CreateTablet for tablet 9704177a036c469ab3d14211271cfd84 (table=default.table_ad3a5796593e44cc8806d94c7b09239a [id=d550ebb246c14e28b1093f5ac04a6cea]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.754631 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9704177a036c469ab3d14211271cfd84. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.756405 25601 ts_tablet_manager.cc:1173] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.757516 25885 raft_consensus.cc:457] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.757773 25885 raft_consensus.cc:2732] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.751s (starting election)
I0619 00:11:54.758016 25885 raft_consensus.cc:479] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.759099 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 234) took 2155ms. Request Metrics: {"mutex_wait_us":33586,"tablet-delete.queue_time_us":1778970}
I0619 00:11:54.759563 25885 leader_election.cc:243] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.759861 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet cb57d196f0724247b6a7f9a34be8377a (table default.table_62055e93837844cd9a5dc2a343a23878 [id=ec5adc7107aa4f16b420ba18f4bc1f63]) successfully deleted
I0619 00:11:54.759994 25885 leader_election.cc:243] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.759874 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 233) took 2157ms. Request Metrics: {"tablet-delete.queue_time_us":2154752}
I0619 00:11:54.760483 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 237) took 2059ms. Request Metrics: {"tablet-delete.queue_time_us":2059118}
W0619 00:11:54.760761 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a because the tablet was not found. No further retry: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:54.763618 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c71886d2a0ac4d869a6806cf551bba7e"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:54.764583 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c71886d2a0ac4d869a6806cf551bba7e in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.766173 25083 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3186954a4b3d40d8be527ce49a270d8b" }
I0619 00:11:54.767556 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 135 ms (attempt = 4)
W0619 00:11:54.767866 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:54.771634 25086 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 19 ms (attempt = 1)
W0619 00:11:54.771885 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:54.776005 25899 tablet.cc:1195] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
W0619 00:11:54.764058 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.774019 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54373917567a479d93dc8a3fd006d2fe" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:54.783942 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f372b9fb174844eca1662c62039a3c71 because the tablet was not found. No further retry: Not found: Tablet not found: f372b9fb174844eca1662c62039a3c71
W0619 00:11:54.785023 25553 leader_election.cc:306] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.786792 25067 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 268 ms (attempt = 5)
W0619 00:11:54.787061 25067 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.786953 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_6ebe326fe1fe4f3580d3f06acc67821f" } new_table_name: "default.table_33f663222f084af580067f0f88d85725" alter_external_catalogs: true
I0619 00:11:54.787317 25423 leader_election.cc:379] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.787556 25423 leader_election.cc:270] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:54.775135 25724 raft_consensus.cc:325] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.788046 25724 raft_consensus.cc:351] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.788038 25847 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.058s (election complete)
I0619 00:11:54.788229 25724 raft_consensus.cc:628] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:54.788281 25847 raft_consensus.cc:2539] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:54.788491 25847 raft_consensus.cc:457] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:54.788717 25847 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.582s (starting election)
I0619 00:11:54.788911 25847 raft_consensus.cc:2784] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.789186 25467 tablet_service.cc:759] Processing CreateTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 (table=default.table_3e0e0119254a4be881b82ecf36174935 [id=a4ae0343c119473d90e61fab1e6df662]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.789803 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b82f9befb9144a7b4be080ac6d92a73. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.792248 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 45 ms (attempt = 2)
W0619 00:11:54.792520 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:54.792979 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:54.793435 25623 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.824s (vote granted)
I0619 00:11:54.774309 26022 tablet.cc:1195] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
I0619 00:11:54.793658 25623 raft_consensus.cc:2245] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:54.789098 25724 consensus_queue.cc:227] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.793637 25600 ts_tablet_manager.cc:1173] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.795195 25598 ts_tablet_manager.cc:1173] T 3e7c4b61f5314e40ad71371155a50b05 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.795290 25847 raft_consensus.cc:479] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:54.796001 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.796298 25847 leader_election.cc:243] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.796706 25847 leader_election.cc:243] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.797966 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 284 ms (attempt = 5)
W0619 00:11:54.798225 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:54.798537 25727 tablet_bootstrap.cc:427] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:54.798882 25727 ts_tablet_manager.cc:1045] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.248s	user 0.006s	sys 0.003s
I0619 00:11:54.800205 25727 raft_consensus.cc:325] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.800705 25084 catalog_manager.cc:3663] Got report from deleted tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:54.800840 25727 raft_consensus.cc:351] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.801383 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:54.801447 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.801573 25727 raft_consensus.cc:628] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:54.802853 25727 consensus_queue.cc:227] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.777557 25325 ts_tablet_manager.cc:1173] T 3e7c4b61f5314e40ad71371155a50b05 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:54.804999 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.805265 25727 ts_tablet_manager.cc:1073] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I0619 00:11:54.805618 25727 ts_tablet_manager.cc:1028] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:54.805668 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.806077 25727 tablet_bootstrap.cc:427] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:54.807216 25084 catalog_manager.cc:3663] Got report from deleted tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:54.807826 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:54.808256 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.806351 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 235) took 2125ms. Request Metrics: {"tablet-delete.queue_time_us":2068883}
W0619 00:11:54.808861 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 70b9c8c4af0945b69b5577e47b61c054 because the tablet was not found. No further retry: Not found: Tablet not found: 70b9c8c4af0945b69b5577e47b61c054
I0619 00:11:54.811719 25084 catalog_manager.cc:3663] Got report from deleted tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:54.812010 25084 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:54.812957 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03522c48c50943b38cae5aa88562e583 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:54.813292 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:54.814649 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.815099 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a33940e5347489993ab4679e9aeafde" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:54.815552 25354 raft_consensus.cc:2784] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.815826 25354 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.701s (vote granted)
I0619 00:11:54.815521 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_10c2abdb4e2f45feb095ecf73709e89b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:54.815865 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_77a5f79a6ba04303b379fc9a17e73f9c"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:54.816432 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_10c2abdb4e2f45feb095ecf73709e89b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.816109 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c5bad852240c4d92bb0cc0cbfccd21c6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:54.816565 26022 tablet.cc:1195] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
W0619 00:11:54.816762 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_77a5f79a6ba04303b379fc9a17e73f9c in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:54.816968 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c5bad852240c4d92bb0cc0cbfccd21c6 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:54.818312 25724 ts_tablet_manager.cc:1073] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.067s	user 0.004s	sys 0.001s
W0619 00:11:54.826115 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.826488 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.808883 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 238) took 2089ms. Request Metrics: {"tablet-delete.queue_time_us":2086195}
I0619 00:11:54.828320 25741 tablet_replica.cc:261] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
W0619 00:11:54.828363 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.828953 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(03522c48c50943b38cae5aa88562e583)
W0619 00:11:54.828955 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0474be659e5e4a4cbfc181b8c812b1eb already in progress: creating tablet
I0619 00:11:54.829221 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:54.829192 25067 catalog_manager.cc:2891] Scheduling retry of 0474be659e5e4a4cbfc181b8c812b1eb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 78 ms (attempt = 2)
I0619 00:11:54.829442 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:54.829685 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:54.830502 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:54.830760 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:54.830971 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(03522c48c50943b38cae5aa88562e583)
I0619 00:11:54.831166 25741 raft_consensus.cc:2042] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:54.831429 25741 consensus_peers.cc:456] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:54.831681 25741 consensus_peers.cc:456] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
I0619 00:11:54.832401 25741 pending_rounds.cc:63] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Trying to abort 1 pending transactions.
I0619 00:11:54.832587 25724 ts_tablet_manager.cc:1028] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:54.832579 25741 pending_rounds.cc:68] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390913551757312 op_type: NO_OP noop_request { }
I0619 00:11:54.832916 25741 raft_consensus.cc:2622] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:54.835160 25724 tablet_bootstrap.cc:427] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:54.836021 25885 raft_consensus.cc:457] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.836287 25885 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.849s (starting election)
I0619 00:11:54.836539 25885 raft_consensus.cc:479] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.836939 25067 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC with a delay of 513 ms (attempt = 6)
W0619 00:11:54.837190 25067 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:54.833930 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 03522c48c50943b38cae5aa88562e583 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.837579 25885 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:54.838004 25885 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.838589 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 70 ms (attempt = 3)
W0619 00:11:54.838877 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:54.839201 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 1034 ms (attempt = 7)
W0619 00:11:54.839432 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c consensus state has no leader
I0619 00:11:54.839586 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a284677c4b7f46e18e25edee6024e3eb" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.813397 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:54.841132 25553 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.836717 25354 raft_consensus.cc:2245] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:54.843559 25086 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 65 ms (attempt = 1)
W0619 00:11:54.843853 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:54.844934 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet b8bc8d4a970843258ab5e9530f425d0f
I0619 00:11:54.845252 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.845700 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.846527 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.847263 25069 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 275 ms (attempt = 5)
W0619 00:11:54.847551 25069 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.847970 25601 tablet_service.cc:800] Processing DeleteTablet for tablet b8bc8d4a970843258ab5e9530f425d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.849975 25325 tablet_service.cc:800] Processing DeleteTablet for tablet b8bc8d4a970843258ab5e9530f425d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.837954 25741 raft_consensus.cc:2069] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:54.851138 25071 catalog_manager.cc:2910] Async tablet task fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fc7686daae274e0dba87703fe6924805 consensus state has no leader
I0619 00:11:54.851678 25736 ts_tablet_manager.cc:1369] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.854122 25730 ts_tablet_manager.cc:1382] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:54.856230 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 543 ms (attempt = 6)
W0619 00:11:54.856467 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:54.859475 25419 leader_election.cc:379] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.855883 25724 tablet_bootstrap.cc:578] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:54.857182 25599 ts_tablet_manager.cc:1173] T a858d8f5709648a3ba7649315ccc9662 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:54.859992 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.862020 25730 log.cc:986] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/7c57c5ca55f24621b66197a3698504bd
I0619 00:11:54.861999 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19ad911ec6704aef85d9b7f172f08bdc" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.859766 25419 leader_election.cc:270] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:54.863760 25847 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.919s (election complete)
I0619 00:11:54.864001 25847 raft_consensus.cc:2539] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:54.864163 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 301 ms (attempt = 5)
W0619 00:11:54.864411 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:54.865471 25847 raft_consensus.cc:593] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:54.865941 25847 consensus_queue.cc:204] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.866472 25730 ts_tablet_manager.cc:1403] T 7c57c5ca55f24621b66197a3698504bd P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:54.834744 25727 tablet_bootstrap.cc:578] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:54.868167 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 172) took 1897ms. Request Metrics: {"mutex_wait_us":1952,"tablet-delete.queue_time_us":1713136}
I0619 00:11:54.869029 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) successfully deleted
I0619 00:11:54.870255 25462 tablet_service.cc:759] Processing CreateTablet for tablet 14179adec4b241be87f42625f5edefc6 (table=default.table_d036ca047e034c39958950ab500d6004 [id=6f8996225b924c6db7ebdeaebb35340e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.870865 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14179adec4b241be87f42625f5edefc6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.872891 25330 ts_tablet_manager.cc:1173] T 9704177a036c469ab3d14211271cfd84 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:54.875572 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.878443 25725 tablet_bootstrap.cc:427] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:54.878823 25725 ts_tablet_manager.cc:1045] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.197s	user 0.006s	sys 0.005s
I0619 00:11:54.879917 25730 tablet_replica.cc:261] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:54.880172 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.880388 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.880580 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.880192 25725 raft_consensus.cc:325] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.880795 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.880967 25725 raft_consensus.cc:351] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.881181 25725 raft_consensus.cc:628] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:54.881819 25725 consensus_queue.cc:227] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.880311 25462 ts_tablet_manager.cc:1173] T 14179adec4b241be87f42625f5edefc6 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.883198 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) successfully deleted
I0619 00:11:54.883637 25725 ts_tablet_manager.cc:1073] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I0619 00:11:54.884017 25725 ts_tablet_manager.cc:1028] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:54.884383 25725 tablet_bootstrap.cc:427] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:54.884555 25068 catalog_manager.cc:2910] Async tablet task 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1dd3f7afe05f49199b18220f9c11ccd2 consensus state has no leader
I0619 00:11:54.886283 25741 ts_tablet_manager.cc:1369] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.888291 25462 tablet_service.cc:759] Processing CreateTablet for tablet 28f9fefa25d14aa089c34c9a0825946e (table=default.table_b2764b59c43146a8ae54b361c93b3406 [id=464afb6b12f4446b89a9682f1d407e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.886159 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a33940e5347489993ab4679e9aeafde" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:54.889194 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 28f9fefa25d14aa089c34c9a0825946e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.889137 25623 raft_consensus.cc:2784] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:54.889377 25623 raft_consensus.cc:2732] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.031s (vote granted)
I0619 00:11:54.890823 25736 ts_tablet_manager.cc:1382] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I0619 00:11:54.891060 25736 log.cc:986] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/36f5eec972454080b7b434b4df7e62e3
I0619 00:11:54.894068 25623 raft_consensus.cc:2245] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:54.895082 25467 ts_tablet_manager.cc:1173] T 2b82f9befb9144a7b4be080ac6d92a73 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.895632 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.897192 25796 raft_consensus.cc:457] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.897428 25796 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.792s (starting election)
I0619 00:11:54.897656 25796 raft_consensus.cc:479] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.898615 25796 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.899022 25796 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.901883 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_1ccfb0785fb045859278f19e637bbdc7" }
I0619 00:11:54.900632 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:54.903393 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 296 ms (attempt = 5)
W0619 00:11:54.903650 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:54.908340 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:54.909255 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.909498 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.909741 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 71 ms (attempt = 2)
I0619 00:11:54.909795 25730 raft_consensus.cc:2042] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
W0619 00:11:54.909993 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:54.910065 25730 raft_consensus.cc:2069] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:54.910382 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 132 ms (attempt = 4)
W0619 00:11:54.910640 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:54.911316 25730 ts_tablet_manager.cc:1369] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:54.916292 25086 catalog_manager.cc:3663] Got report from deleted tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:54.916966 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:54.917448 25724 tablet_bootstrap.cc:427] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:54.917829 25724 ts_tablet_manager.cc:1045] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.083s	user 0.004s	sys 0.004s
I0619 00:11:54.919044 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.919643 25724 raft_consensus.cc:325] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.920275 25724 raft_consensus.cc:351] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.920476 25724 raft_consensus.cc:628] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:54.921022 25724 consensus_queue.cc:227] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.924178 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.932291 25086 master_service.cc:523] ReplaceTablet: received request to replace tablet d0624684b6b04ba3be260b2705dc00e1 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:54.937638 25736 ts_tablet_manager.cc:1403] T 36f5eec972454080b7b434b4df7e62e3 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:54.945016 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet ee4c4769f75141f9b2ef864ab8a8d9e0 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:54.945256 25423 outbound_call.cc:287] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.24.119.61:46325, user_credentials={real_user=jenkins}} blocked reactor thread for 59486.4us
I0619 00:11:54.952306 25325 tablet_service.cc:759] Processing CreateTablet for tablet 309a14ee0658485b9ea724d07b8cce20 (table=default.table_d9245875d6214b88a63997a3d300f512 [id=1e349cc1896d47d6832c30fc440e54aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.952927 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 309a14ee0658485b9ea724d07b8cce20. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.953483 25330 tablet_service.cc:759] Processing CreateTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c (table=default.table_54575c07db144677bb4aeb1144a588b5 [id=5f0526a7014549d6b2915f6f85d03005]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.954138 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0ef73ff0abf44e5aa66cee31180c083c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.956989 25725 tablet_bootstrap.cc:578] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:54.958653 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.959537 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet d0624684b6b04ba3be260b2705dc00e1
I0619 00:11:54.959883 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:11:54.960273 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:11:54.960844 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:11:54.961026 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54373917567a479d93dc8a3fd006d2fe" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:54.960793 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet f1b566b2b58f4955aa25ae990a4a8a1a
I0619 00:11:54.961644 25331 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.961797 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.962204 25086 catalog_manager.cc:4596] ReplaceTablet: tablet d0624684b6b04ba3be260b2705dc00e1 deleted and replaced by tablet c2cf56dada3c41d4920a4091a5fdfded
I0619 00:11:54.962172 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:54.962875 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:11:54.964469 25419 leader_election.cc:306] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.965092 25331 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:54.951612 25599 tablet_service.cc:759] Processing CreateTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c (table=default.table_54575c07db144677bb4aeb1144a588b5 [id=5f0526a7014549d6b2915f6f85d03005]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.965718 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0ef73ff0abf44e5aa66cee31180c083c. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:54.962852 25558 leader_election.cc:306] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.965922 25558 leader_election.cc:270] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.966562 25885 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.616s (election complete)
I0619 00:11:54.966804 25885 raft_consensus.cc:2487] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.964262 25598 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.967025 25727 tablet_bootstrap.cc:427] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:54.967339 25727 ts_tablet_manager.cc:1045] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.161s	user 0.004s	sys 0.002s
I0619 00:11:54.968716 25727 raft_consensus.cc:325] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.969301 25727 raft_consensus.cc:351] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:54.969401 25598 tablet_service.cc:759] Processing CreateTablet for tablet 309a14ee0658485b9ea724d07b8cce20 (table=default.table_d9245875d6214b88a63997a3d300f512 [id=1e349cc1896d47d6832c30fc440e54aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.969962 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 309a14ee0658485b9ea724d07b8cce20. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.961872 25601 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:54.970825 25331 tablet_service.cc:759] Processing CreateTablet for tablet 455d8e1ee5be484b8005312d2d886821 (table=default.table_58781b6793a04331a5a69cdbb7cd2cc4 [id=f7588ecc502740d1bb371211ac284938]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.975350 25601 tablet_service.cc:759] Processing CreateTablet for tablet 455d8e1ee5be484b8005312d2d886821 (table=default.table_58781b6793a04331a5a69cdbb7cd2cc4 [id=f7588ecc502740d1bb371211ac284938]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.975278 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c74748783ca3450eb3c5c3397e03d506" } new_table_name: "default.table_3ad5f49b495d4a738b79181172cfd9f4" alter_external_catalogs: true
I0619 00:11:54.975980 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 455d8e1ee5be484b8005312d2d886821. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:54.979243 25336 tablet_service.cc:759] Processing CreateTablet for tablet bae6d308dae54b0394a364d2988ec6b3 (table=default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.979295 25598 ts_tablet_manager.cc:1173] T 309a14ee0658485b9ea724d07b8cce20 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.981798 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 83 ms (attempt = 3)
W0619 00:11:54.982064 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:54.986708 25724 ts_tablet_manager.cc:1073] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.069s	user 0.003s	sys 0.000s
I0619 00:11:54.987054 25724 ts_tablet_manager.cc:1028] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
W0619 00:11:54.987421 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.987970 25724 tablet_bootstrap.cc:427] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:54.988970 25558 leader_election.cc:306] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:54.966037 25467 tablet_service.cc:759] Processing CreateTablet for tablet dba1210d9c22454f8c9e0b0293583f0c (table=default.table_c74748783ca3450eb3c5c3397e03d506 [id=040c5029d5054cc48c9f066058cb6a45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:54.989243 25947 raft_consensus.cc:457] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:54.986171 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.989516 25947 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.580s (starting election)
I0619 00:11:54.989959 25558 leader_election.cc:270] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:54.989974 25947 raft_consensus.cc:479] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.990964 25885 raft_consensus.cc:2732] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.965s (election complete)
I0619 00:11:54.990967 25947 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:54.991184 25885 raft_consensus.cc:2487] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:54.991355 25947 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:54.989313 25470 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:54.987094 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a284677c4b7f46e18e25edee6024e3eb" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:54.992035 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.969471 25727 raft_consensus.cc:628] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
W0619 00:11:54.992713 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.992816 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:54.993268 25623 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.957s (vote granted)
I0619 00:11:54.993481 25623 raft_consensus.cc:2245] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:54.993085 25727 consensus_queue.cc:227] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:54.995347 25727 ts_tablet_manager.cc:1073] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.028s	user 0.003s	sys 0.001s
I0619 00:11:54.995605 25601 ts_tablet_manager.cc:1173] T 455d8e1ee5be484b8005312d2d886821 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:54.996886 25086 catalog_manager.cc:3786] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:55.000986 25599 ts_tablet_manager.cc:1173] T 0ef73ff0abf44e5aa66cee31180c083c P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:54.989485 25419 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.1:32951, user_credentials={real_user=jenkins}} blocked reactor thread for 86378.8us
I0619 00:11:55.004604 25601 tablet_service.cc:759] Processing CreateTablet for tablet bae6d308dae54b0394a364d2988ec6b3 (table=default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.005261 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bae6d308dae54b0394a364d2988ec6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.007661 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dba1210d9c22454f8c9e0b0293583f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.002424 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 218) took 2349ms. Request Metrics: {"tablet-delete.queue_time_us":2049243}
I0619 00:11:55.008222 25736 tablet_replica.cc:261] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:55.008464 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:54.988922 25423 leader_election.cc:379] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.008702 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.008765 25423 leader_election.cc:270] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:55.009114 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.009266 25796 raft_consensus.cc:2732] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.992s (election complete)
I0619 00:11:55.009376 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.009526 25796 raft_consensus.cc:2539] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:55.009896 25796 raft_consensus.cc:593] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:55.009902 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.010401 25796 consensus_queue.cc:204] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.010428 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.013422 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(775f0b6b309c49278066a5e7cf623af2)
W0619 00:11:54.995699 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.020987 25470 tablet_service.cc:759] Processing CreateTablet for tablet fa0ab4b6eef04ae08eb8994b49ca531e (table=default.table_6ebe326fe1fe4f3580d3f06acc67821f [id=b9bfdc8e55474269827835656bfecf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.021569 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fa0ab4b6eef04ae08eb8994b49ca531e. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:55.022508 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.022920 25725 tablet_bootstrap.cc:427] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:55.023401 25725 ts_tablet_manager.cc:1045] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.139s	user 0.006s	sys 0.001s
I0619 00:11:55.025290 25467 ts_tablet_manager.cc:1173] T dba1210d9c22454f8c9e0b0293583f0c P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.026803 25730 ts_tablet_manager.cc:1382] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:55.027465 25730 log.cc:986] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/775f0b6b309c49278066a5e7cf623af2
I0619 00:11:55.030402 25730 ts_tablet_manager.cc:1403] T 775f0b6b309c49278066a5e7cf623af2 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:11:55.031875 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:54.968984 25325 ts_tablet_manager.cc:1173] T 309a14ee0658485b9ea724d07b8cce20 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.033064 25330 ts_tablet_manager.cc:1173] T 0ef73ff0abf44e5aa66cee31180c083c P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.028741 25725 raft_consensus.cc:325] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.034785 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4c653a255984d3bac88df63362b9334" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.035542 25727 ts_tablet_manager.cc:1028] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:55.035974 25727 tablet_bootstrap.cc:427] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:55.037503 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bae6d308dae54b0394a364d2988ec6b3. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:55.038334 25419 leader_election.cc:306] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.039185 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 455d8e1ee5be484b8005312d2d886821. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.039556 25736 raft_consensus.cc:2042] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
W0619 00:11:55.039790 25558 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.040067 25558 leader_election.cc:270] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.040134 25736 consensus_peers.cc:456] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:55.040339 25736 consensus_peers.cc:456] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:55.040695 25736 pending_rounds.cc:63] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Trying to abort 1 pending transactions.
I0619 00:11:55.040844 25736 pending_rounds.cc:68] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390922288902144 op_type: NO_OP noop_request { }
I0619 00:11:55.041121 25736 raft_consensus.cc:2622] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:55.041366 25885 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.147s (election complete)
W0619 00:11:55.041241 25423 consensus_peers.cc:434] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 397ae10bb6d841b4ba561ec431d29864. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:55.041584 25885 raft_consensus.cc:2487] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.042001 25736 raft_consensus.cc:2069] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.043751 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 292 ms (attempt = 5)
W0619 00:11:55.044026 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:55.044109 25940 raft_consensus.cc:457] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.044354 25940 raft_consensus.cc:2732] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.653s (starting election)
I0619 00:11:55.044589 25940 raft_consensus.cc:479] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:55.044924 26043 consensus_queue.cc:1042] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "10bda0b4856f4441b5aed24cdc4764a8" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I0619 00:11:55.045575 25940 leader_election.cc:243] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.045661 25727 tablet_bootstrap.cc:578] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:55.046030 25940 leader_election.cc:243] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.046514 25468 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.047948 25736 ts_tablet_manager.cc:1369] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.048326 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_fb89f806be98424aa43bd01b3f816a80" } new_table_name: "default.table_ec6a5ef017d74c7eb1777f1bf058a3da" alter_external_catalogs: true
I0619 00:11:55.048372 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_dc31a7293c29471394747b8090909823"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.049547 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dc31a7293c29471394747b8090909823 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.049450 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_4f232715babf40febb76a8321c6658c8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.050330 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4f232715babf40febb76a8321c6658c8 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:55.050514 25290 leader_election.cc:306] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:55.050628 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.051376 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_2e052476ba2c4ab08ef5774928db6084" }
W0619 00:11:55.052507 25423 consensus_peers.cc:434] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 7c57c5ca55f24621b66197a3698504bd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:55.053200 25725 raft_consensus.cc:351] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.053640 25725 raft_consensus.cc:628] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:55.055768 25067 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 520 ms (attempt = 6)
W0619 00:11:55.056035 25067 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:55.032124 25470 ts_tablet_manager.cc:1173] T fa0ab4b6eef04ae08eb8994b49ca531e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.054613 25725 consensus_queue.cc:227] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.056972 25462 ts_tablet_manager.cc:1173] T 28f9fefa25d14aa089c34c9a0825946e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.056913 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 175) took 1802ms. Request Metrics: {"tablet-delete.queue_time_us":1617565}
I0619 00:11:55.057793 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 775f0b6b309c49278066a5e7cf623af2 (table default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]) successfully deleted
I0619 00:11:55.058521 25336 ts_tablet_manager.cc:1173] T bae6d308dae54b0394a364d2988ec6b3 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:55.059211 25290 leader_election.cc:306] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:55.059456 25290 leader_election.cc:270] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.059911 25940 raft_consensus.cc:2732] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.540s (election complete)
I0619 00:11:55.060127 25940 raft_consensus.cc:2487] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.062749 25724 tablet_bootstrap.cc:578] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:55.062959 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_5e0e10c382ce4a0b8aeb14cc019630a0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.063798 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5e0e10c382ce4a0b8aeb14cc019630a0 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.064800 25082 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 48 ms (attempt = 1)
W0619 00:11:55.065104 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:55.065670 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_042bb1895e44487bbf4563658402bf87"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.066637 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_042bb1895e44487bbf4563658402bf87 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.066787 25084 catalog_manager.cc:3663] Got report from deleted tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:55.066942 25080 catalog_manager.cc:3663] Got report from deleted tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:55.067426 25725 ts_tablet_manager.cc:1073] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.044s	user 0.004s	sys 0.000s
I0619 00:11:55.067706 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.067791 25725 ts_tablet_manager.cc:1028] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:55.068965 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 154 ms (attempt = 4)
W0619 00:11:55.069226 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:55.071393 25725 tablet_bootstrap.cc:427] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:55.072052 25084 catalog_manager.cc:3663] Got report from deleted tablet d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
W0619 00:11:55.072326 25067 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:55.072332 25084 catalog_manager.cc:3663] Got report from deleted tablet 775f0b6b309c49278066a5e7cf623af2 (table default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:55.073362 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7c57c5ca55f24621b66197a3698504bd on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.073746 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.074091 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:55.075515 25080 catalog_manager.cc:3663] Got report from deleted tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:55.076151 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4a88e394b199431e8b794a1b1aec847f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.078959 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.080948 25885 raft_consensus.cc:457] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.081218 25885 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.520s (starting election)
I0619 00:11:55.081495 25885 raft_consensus.cc:479] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.082527 25885 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.082952 25885 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.083247 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 557 ms (attempt = 6)
W0619 00:11:55.083509 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:55.085160 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 0a33940e5347489993ab4679e9aeafde
I0619 00:11:55.085472 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.085508 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.085913 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.086272 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
W0619 00:11:55.086948 25553 leader_election.cc:306] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.087894 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.090224 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:55.090723 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.091688 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.092985 25487 raft_consensus.cc:1081] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:55.093524 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.094477 25940 consensus_queue.cc:963] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:55.058605 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 174) took 1804ms. Request Metrics: {"tablet-delete.queue_time_us":1796033}
I0619 00:11:55.096966 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.097354 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:55.097823 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:55.098742 25558 leader_election.cc:306] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.099007 25558 leader_election.cc:270] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.099572 25467 tablet_service.cc:759] Processing CreateTablet for tablet 8b437c976cfd41aa8f2f2f38b3762005 (table=default.table_e874b4b421a446c48acc51dc98a14038 [id=48e78cf9fa7840a3b2b84621f7c06cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.099845 25885 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.714s (election complete)
I0619 00:11:55.100054 25885 raft_consensus.cc:2487] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.100203 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8b437c976cfd41aa8f2f2f38b3762005. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.101361 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 180) took 1789ms. Request Metrics: {"tablet-delete.queue_time_us":1782090}
I0619 00:11:55.102823 25741 ts_tablet_manager.cc:1382] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:55.103457 25601 ts_tablet_manager.cc:1173] T bae6d308dae54b0394a364d2988ec6b3 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.104904 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 183) took 1665ms. Request Metrics: {"tablet-delete.queue_time_us":1646456}
I0619 00:11:55.105422 25724 tablet_bootstrap.cc:427] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:55.106339 25724 ts_tablet_manager.cc:1045] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.118s	user 0.005s	sys 0.006s
I0619 00:11:55.104367 25331 ts_tablet_manager.cc:1173] T 455d8e1ee5be484b8005312d2d886821 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.107985 25724 raft_consensus.cc:325] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.108619 25724 raft_consensus.cc:351] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.108469 25741 log.cc:986] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/03522c48c50943b38cae5aa88562e583
I0619 00:11:55.108870 25724 raft_consensus.cc:628] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:55.109568 25741 ts_tablet_manager.cc:1403] T 03522c48c50943b38cae5aa88562e583 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:55.109285 25724 consensus_queue.cc:227] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.112269 25725 tablet_bootstrap.cc:578] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:55.107107 25730 tablet_replica.cc:261] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:55.112926 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(54373917567a479d93dc8a3fd006d2fe)
W0619 00:11:55.110199 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:55.111224 25290 consensus_peers.cc:434] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet a85577b0beba4276b963e5c360544bc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:55.113131 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:55.111533 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 4a88e394b199431e8b794a1b1aec847f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.114871 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 65 ms (attempt = 2)
W0619 00:11:55.115135 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:55.116205 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:55.116468 25727 tablet_bootstrap.cc:427] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:55.116804 25727 ts_tablet_manager.cc:1045] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.081s	user 0.006s	sys 0.001s
I0619 00:11:55.117039 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:55.118049 25727 raft_consensus.cc:325] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.118667 25727 raft_consensus.cc:351] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.118878 25727 raft_consensus.cc:628] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:55.119977 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:55.120347 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:55.118237 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 239) took 2368ms. Request Metrics: {"spinlock_wait_cycles":4480,"tablet-delete.queue_time_us":2073618}
W0619 00:11:55.120411 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 775f0b6b309c49278066a5e7cf623af2 because the tablet was not found. No further retry: Not found: Tablet not found: 775f0b6b309c49278066a5e7cf623af2
I0619 00:11:55.120561 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:55.120774 25730 raft_consensus.cc:2042] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
W0619 00:11:55.121085 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times.
W0619 00:11:55.122228 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 6040e369e0fb445fb993d12dd883039b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6040e369e0fb445fb993d12dd883039b already in progress: creating tablet
I0619 00:11:55.122619 25071 catalog_manager.cc:2891] Scheduling retry of 6040e369e0fb445fb993d12dd883039b Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 30 ms (attempt = 1)
I0619 00:11:55.123093 25069 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 549 ms (attempt = 6)
W0619 00:11:55.123780 25069 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
W0619 00:11:55.124557 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb because the tablet was not found. No further retry: Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:55.127604 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.129460 25462 tablet_service.cc:759] Processing CreateTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 (table=default.table_30bcafcc4a6f46e99005fd47447e7dbe [id=6ba1ca844e8d4a9ab4157314c5e4f543]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.134418 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3e7c4b61f5314e40ad71371155a50b05. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.141814 25468 tablet_service.cc:759] Processing CreateTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 (table=default.table_53e73552aa6d4161a595d779a9fc96cc [id=1697152e5d9f409ab68219b93b01250f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.142357 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8ca3a0771b9e41eeb0d14d886f970159. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.142652 25724 ts_tablet_manager.cc:1073] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.036s	user 0.004s	sys 0.000s
W0619 00:11:55.143240 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.144326 25730 raft_consensus.cc:2069] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:55.145110 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:55.145838 25069 catalog_manager.cc:2910] Async tablet task f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f1b566b2b58f4955aa25ae990a4a8a1a consensus state has no leader
I0619 00:11:55.146242 25087 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:55.146932 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:55.150800 25086 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:55.151762 26052 tablet.cc:1195] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
I0619 00:11:55.152014 25087 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:55.152575 25468 ts_tablet_manager.cc:1173] T 8ca3a0771b9e41eeb0d14d886f970159 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.152604 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:55.153290 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:55.154505 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
W0619 00:11:55.156162 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.155591 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.152659 25730 ts_tablet_manager.cc:1369] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:55.159683 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.160120 25087 catalog_manager.cc:3663] Got report from deleted tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:55.160784 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 397ae10bb6d841b4ba561ec431d29864 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:55.163244 25086 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:55.163869 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:55.166309 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 526 ms (attempt = 6)
W0619 00:11:55.166553 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:55.168710 25724 ts_tablet_manager.cc:1028] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:55.172546 25724 tablet_bootstrap.cc:427] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:55.173012 25423 leader_election.cc:379] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.173378 25423 leader_election.cc:270] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:55.173924 25847 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.645s (election complete)
I0619 00:11:55.174111 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet 775f0b6b309c49278066a5e7cf623af2 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:55.174166 25847 raft_consensus.cc:2539] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:55.174394 25847 raft_consensus.cc:457] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:55.174589 25847 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.701s (starting election)
I0619 00:11:55.174813 25847 raft_consensus.cc:2784] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.174890 25724 tablet_bootstrap.cc:578] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:55.174994 26052 tablet.cc:1195] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
I0619 00:11:55.176928 25086 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_54575c07db144677bb4aeb1144a588b5" }
W0619 00:11:55.132601 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:55.134378 25290 consensus_peers.cc:434] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 7f60ea5c5def47ee846d0186bbc90c18. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18. Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:11:55.177959 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 43641.9us
I0619 00:11:55.119264 25727 consensus_queue.cc:227] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.180658 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 71 ms (attempt = 3)
W0619 00:11:55.180960 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:55.180269 25847 raft_consensus.cc:479] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.181313 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 03522c48c50943b38cae5aa88562e583 (table default.table_0a2ef368d0f945eaaeeb3e58d57ef718 [id=3af4854117e94c6a9ac3516c131ab144]) successfully deleted
I0619 00:11:55.181634 25847 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:55.183127 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.183549 25847 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.183706 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:55.184450 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.184835 25727 ts_tablet_manager.cc:1073] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.068s	user 0.005s	sys 0.000s
W0619 00:11:55.185042 25419 leader_election.cc:306] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.185209 25727 ts_tablet_manager.cc:1028] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:55.184120 25336 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.185636 25727 tablet_bootstrap.cc:427] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:55.185981 25081 catalog_manager.cc:3663] Got report from deleted tablet 775f0b6b309c49278066a5e7cf623af2 (table default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:55.186655 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 775f0b6b309c49278066a5e7cf623af2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:55.189142 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.179849 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.189764 25468 tablet_service.cc:759] Processing CreateTablet for tablet a858d8f5709648a3ba7649315ccc9662 (table=default.table_27e7515742a84639bd10967f26b8d93f [id=2f4c8afb95b945138a65de871be4b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.190042 25741 tablet_replica.cc:261] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:55.190310 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.190558 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.190964 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.191318 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.191582 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.191844 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.192073 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(775f0b6b309c49278066a5e7cf623af2)
I0619 00:11:55.194511 25725 tablet_bootstrap.cc:427] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:55.194891 25725 ts_tablet_manager.cc:1045] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.124s	user 0.005s	sys 0.001s
I0619 00:11:55.196539 25725 raft_consensus.cc:325] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.197187 25725 raft_consensus.cc:351] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.197391 25725 raft_consensus.cc:628] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:55.197842 25725 consensus_queue.cc:227] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.198477 25736 ts_tablet_manager.cc:1382] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:55.199849 25736 log.cc:986] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/775f0b6b309c49278066a5e7cf623af2
I0619 00:11:55.200731 25467 ts_tablet_manager.cc:1173] T 8b437c976cfd41aa8f2f2f38b3762005 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.200758 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 513 ms (attempt = 6)
W0619 00:11:55.201020 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:55.203560 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a858d8f5709648a3ba7649315ccc9662. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.211551 25727 tablet_bootstrap.cc:578] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:55.212976 25736 ts_tablet_manager.cc:1403] T 775f0b6b309c49278066a5e7cf623af2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:55.214787 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 219) took 2561ms. Request Metrics: {"spinlock_wait_cycles":62037888,"tablet-delete.queue_time_us":2354244}
I0619 00:11:55.192276 25741 raft_consensus.cc:2042] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:55.216852 25741 raft_consensus.cc:2069] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.218688 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:55.219164 25623 raft_consensus.cc:2784] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.219436 25623 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.811s (vote granted)
I0619 00:11:55.219794 25724 tablet_bootstrap.cc:427] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:55.220146 25724 ts_tablet_manager.cc:1045] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.048s	user 0.004s	sys 0.000s
W0619 00:11:55.222084 25071 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:55.221554 25724 raft_consensus.cc:325] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.222791 25724 raft_consensus.cc:351] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.221573 25741 ts_tablet_manager.cc:1369] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.223004 25724 raft_consensus.cc:628] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:55.223829 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 280 ms (attempt = 5)
W0619 00:11:55.224081 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:55.217353 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:55.217816 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.223870 25724 consensus_queue.cc:227] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.225996 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 0ef73ff0abf44e5aa66cee31180c083c
I0619 00:11:55.199540 25725 ts_tablet_manager.cc:1073] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I0619 00:11:55.218444 25423 leader_election.cc:379] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:55.229404 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.229817 25725 ts_tablet_manager.cc:1028] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:55.230139 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.231544 25467 tablet_service.cc:759] Processing CreateTablet for tablet 9704177a036c469ab3d14211271cfd84 (table=default.table_ad3a5796593e44cc8806d94c7b09239a [id=d550ebb246c14e28b1093f5ac04a6cea]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.233104 25423 leader_election.cc:379] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.233419 25423 leader_election.cc:270] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:55.234313 25847 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.876s (election complete)
I0619 00:11:55.234552 25847 raft_consensus.cc:2539] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:55.234799 25847 raft_consensus.cc:457] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:55.235038 25847 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.794s (starting election)
I0619 00:11:55.235304 25847 raft_consensus.cc:2784] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.235097 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9704177a036c469ab3d14211271cfd84. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:55.235836 25290 consensus_peers.cc:434] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet a85577b0beba4276b963e5c360544bc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7. Retrying in the next heartbeat period. Already tried 2 times.
W0619 00:11:55.236152 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.236965 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 2058 ms (attempt = 8)
W0619 00:11:55.237216 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:55.239739 25885 raft_consensus.cc:457] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.239984 25885 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.993s (starting election)
I0619 00:11:55.240213 25885 raft_consensus.cc:479] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.241135 25885 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.241924 25885 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.243492 25736 tablet_replica.cc:261] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:55.243784 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:55.243984 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:55.243798 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "981d862d1bab4b14bb10e6fc2174e1cc" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.244170 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:55.244364 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:55.244606 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:55.244820 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:55.245005 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:55.245162 25736 raft_consensus.cc:2042] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:55.245339 25736 consensus_peers.cc:456] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
W0619 00:11:55.245283 25553 leader_election.cc:306] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.245522 25736 consensus_peers.cc:456] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:55.245911 25736 raft_consensus.cc:2069] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.247202 25725 tablet_bootstrap.cc:427] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:55.249598 25847 raft_consensus.cc:479] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.250931 25623 raft_consensus.cc:2245] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
W0619 00:11:55.251420 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.251767 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0ef73ff0abf44e5aa66cee31180c083c on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.252022 25087 catalog_manager.cc:3663] Got report from deleted tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) (Table deleted at 2018-06-19 00:11:49 PDT): Sending delete request for this tablet
I0619 00:11:55.252286 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0ef73ff0abf44e5aa66cee31180c083c on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.252714 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0ef73ff0abf44e5aa66cee31180c083c on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.254194 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.255228 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 142 ms (attempt = 4)
W0619 00:11:55.255481 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:55.256973 25947 raft_consensus.cc:457] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.257086 25601 tablet_service.cc:759] Processing CreateTablet for tablet 0fee3be144c14edab6db58c99e3cda7d (table=default.table_c71886d2a0ac4d869a6806cf551bba7e [id=e4d5113658b04af49f2b7d2e67cf71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.257201 25947 raft_consensus.cc:2732] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.739s (starting election)
I0619 00:11:55.252689 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a85577b0beba4276b963e5c360544bc7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:49 PDT)
I0619 00:11:55.257644 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0fee3be144c14edab6db58c99e3cda7d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.257443 25947 raft_consensus.cc:479] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.258137 25330 tablet_service.cc:759] Processing CreateTablet for tablet ec8f89d6855544a584eb2ebcd88b7fb6 (table=default.table_77a5f79a6ba04303b379fc9a17e73f9c [id=e81ccf2cf02a4d278717d5fd8274f2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.258389 25947 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.258745 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ec8f89d6855544a584eb2ebcd88b7fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.258787 25947 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.259610 25336 tablet_service.cc:759] Processing CreateTablet for tablet 0fee3be144c14edab6db58c99e3cda7d (table=default.table_c71886d2a0ac4d869a6806cf551bba7e [id=e4d5113658b04af49f2b7d2e67cf71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.260304 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.262140 25331 tablet_service.cc:800] Processing DeleteTablet for tablet a85577b0beba4276b963e5c360544bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:49 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.262903 25599 tablet_service.cc:759] Processing CreateTablet for tablet ec8f89d6855544a584eb2ebcd88b7fb6 (table=default.table_77a5f79a6ba04303b379fc9a17e73f9c [id=e81ccf2cf02a4d278717d5fd8274f2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.262972 25724 ts_tablet_manager.cc:1073] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.043s	user 0.004s	sys 0.000s
I0619 00:11:55.263440 25724 ts_tablet_manager.cc:1028] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:55.263494 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ec8f89d6855544a584eb2ebcd88b7fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.263849 25724 tablet_bootstrap.cc:427] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:55.264370 25727 tablet_bootstrap.cc:427] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:55.264730 25727 ts_tablet_manager.cc:1045] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.079s	user 0.006s	sys 0.000s
I0619 00:11:55.266086 25727 raft_consensus.cc:325] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.266696 25727 raft_consensus.cc:351] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.266887 25727 raft_consensus.cc:628] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
W0619 00:11:55.267714 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.267309 25727 consensus_queue.cc:227] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.269752 25727 ts_tablet_manager.cc:1073] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I0619 00:11:55.270550 25727 ts_tablet_manager.cc:1028] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:55.270956 25727 tablet_bootstrap.cc:427] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:55.272775 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0fee3be144c14edab6db58c99e3cda7d. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:55.275164 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.274297 25847 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.275599 25847 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.275985 25331 tablet_service.cc:759] Processing CreateTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 (table=default.table_c5bad852240c4d92bb0cc0cbfccd21c6 [id=fca6fa877b5f4226858ebe70c5d90418]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.276168 25598 tablet_service.cc:759] Processing CreateTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 (table=default.table_c5bad852240c4d92bb0cc0cbfccd21c6 [id=fca6fa877b5f4226858ebe70c5d90418]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.276787 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eb36a22cf80841b58fd5ce9e6998bf00. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.277007 25079 catalog_manager.cc:3663] Got report from deleted tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d): Sending delete request for this tablet
I0619 00:11:55.277701 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:55.277995 25599 ts_tablet_manager.cc:1173] T ec8f89d6855544a584eb2ebcd88b7fb6 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.278599 25600 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.279507 25081 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_4f232715babf40febb76a8321c6658c8" } new_table_name: "default.table_718ed0cda54b4ab8b71539f7f044d28f" alter_external_catalogs: true
I0619 00:11:55.281728 25079 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_5f1d24330f9f41969881d224929b3ee2" } new_table_name: "default.table_9a8c9964bcff4effb2b624ef5bfdef1c" alter_external_catalogs: true
I0619 00:11:55.285570 25468 ts_tablet_manager.cc:1173] T a858d8f5709648a3ba7649315ccc9662 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.286983 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 775f0b6b309c49278066a5e7cf623af2 (table default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]) successfully deleted
I0619 00:11:55.287665 25724 tablet_bootstrap.cc:578] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:55.287983 25423 leader_election.cc:306] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
I0619 00:11:55.288250 25423 leader_election.cc:270] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.289739 25847 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.022s (election complete)
I0619 00:11:55.289974 25847 raft_consensus.cc:2487] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.290778 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eb36a22cf80841b58fd5ce9e6998bf00. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.291052 25470 tablet_service.cc:759] Processing CreateTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c (table=default.table_54575c07db144677bb4aeb1144a588b5 [id=5f0526a7014549d6b2915f6f85d03005]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.291667 25464 tablet_service.cc:759] Processing CreateTablet for tablet 309a14ee0658485b9ea724d07b8cce20 (table=default.table_d9245875d6214b88a63997a3d300f512 [id=1e349cc1896d47d6832c30fc440e54aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.291671 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0ef73ff0abf44e5aa66cee31180c083c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.292356 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 309a14ee0658485b9ea724d07b8cce20. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.293280 25624 raft_consensus.cc:1081] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:55.293987 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:55.294420 25623 raft_consensus.cc:2784] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.294673 25623 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.042s (vote granted)
I0619 00:11:55.299607 25468 tablet_service.cc:800] Processing DeleteTablet for tablet b8bc8d4a970843258ab5e9530f425d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.300149 25081 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 54 ms (attempt = 1)
W0619 00:11:55.300596 25081 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 has no consensus state
I0619 00:11:55.302098 25467 ts_tablet_manager.cc:1173] T 9704177a036c469ab3d14211271cfd84 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.303772 25601 ts_tablet_manager.cc:1173] T 0fee3be144c14edab6db58c99e3cda7d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.304441 25727 tablet_bootstrap.cc:578] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:55.291646 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "981d862d1bab4b14bb10e6fc2174e1cc" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.307314 25598 ts_tablet_manager.cc:1173] T eb36a22cf80841b58fd5ce9e6998bf00 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.307474 25599 tablet_service.cc:759] Processing CreateTablet for tablet c2cf56dada3c41d4920a4091a5fdfded (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.308140 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c2cf56dada3c41d4920a4091a5fdfded. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.311844 25623 raft_consensus.cc:2245] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:55.314409 25079 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC with a delay of 59 ms (attempt = 1)
W0619 00:11:55.314714 25079 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:55.314826 25741 ts_tablet_manager.cc:1382] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:55.315060 25741 log.cc:986] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/775f0b6b309c49278066a5e7cf623af2
I0619 00:11:55.319799 25470 ts_tablet_manager.cc:1173] T 0ef73ff0abf44e5aa66cee31180c083c P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.320253 25730 ts_tablet_manager.cc:1382] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:55.320480 25462 ts_tablet_manager.cc:1173] T 3e7c4b61f5314e40ad71371155a50b05 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.320514 25730 log.cc:986] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/54373917567a479d93dc8a3fd006d2fe
I0619 00:11:55.321069 25730 ts_tablet_manager.cc:1403] T 54373917567a479d93dc8a3fd006d2fe P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:55.321722 25736 ts_tablet_manager.cc:1369] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.323333 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 182) took 1925ms. Request Metrics: {"mutex_wait_us":23242,"tablet-delete.queue_time_us":1646696}
I0619 00:11:55.324142 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 54373917567a479d93dc8a3fd006d2fe (table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]) successfully deleted
I0619 00:11:55.324286 25730 tablet_replica.cc:261] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:55.325073 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:55.325305 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:55.325517 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:55.325641 25464 ts_tablet_manager.cc:1173] T 309a14ee0658485b9ea724d07b8cce20 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.325938 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:55.329071 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:55.329354 25601 tablet_service.cc:759] Processing CreateTablet for tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c (table=default.table_10c2abdb4e2f45feb095ecf73709e89b [id=191ee261cfbe4ca3b04553067de4e26d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.329985 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.334640 25725 tablet_bootstrap.cc:578] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:55.333622 25724 tablet_bootstrap.cc:427] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:55.335158 25724 ts_tablet_manager.cc:1045] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.071s	user 0.005s	sys 0.000s
I0619 00:11:55.336042 25599 ts_tablet_manager.cc:1173] T c2cf56dada3c41d4920a4091a5fdfded P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.336091 25464 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.337117 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 538 ms (attempt = 6)
W0619 00:11:55.337232 25558 leader_election.cc:306] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:55.337400 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:55.337496 25558 leader_election.cc:270] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.337857 25464 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.337887 25462 tablet_service.cc:759] Processing CreateTablet for tablet 455d8e1ee5be484b8005312d2d886821 (table=default.table_58781b6793a04331a5a69cdbb7cd2cc4 [id=f7588ecc502740d1bb371211ac284938]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.338519 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 455d8e1ee5be484b8005312d2d886821. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.338631 25885 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.745s (election complete)
I0619 00:11:55.338850 25885 raft_consensus.cc:2487] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:55.339095 25423 consensus_peers.cc:434] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 7c57c5ca55f24621b66197a3698504bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c57c5ca55f24621b66197a3698504bd. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:55.336805 25724 raft_consensus.cc:325] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.340083 25724 raft_consensus.cc:351] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.340266 25724 raft_consensus.cc:628] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:55.329319 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:55.340765 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:55.340996 25730 raft_consensus.cc:2042] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:55.341276 25730 raft_consensus.cc:2069] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.342440 25730 ts_tablet_manager.cc:1369] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.343083 25885 raft_consensus.cc:457] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.343315 25885 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.914s (starting election)
I0619 00:11:55.343551 25885 raft_consensus.cc:479] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.344516 25885 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.344921 25885 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:55.350994 25067 catalog_manager.cc:2910] Async tablet task b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet b8bc8d4a970843258ab5e9530f425d0f consensus state has no leader
I0619 00:11:55.355760 25071 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 33 ms (attempt = 2)
W0619 00:11:55.356060 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 has no consensus state
I0619 00:11:55.361004 25947 raft_consensus.cc:457] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.361289 25947 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.239s (starting election)
I0619 00:11:55.361539 25947 raft_consensus.cc:479] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.363945 25947 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.364385 25947 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.315518 25741 ts_tablet_manager.cc:1403] T 775f0b6b309c49278066a5e7cf623af2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:55.365195 25464 tablet_service.cc:759] Processing CreateTablet for tablet bae6d308dae54b0394a364d2988ec6b3 (table=default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.365820 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bae6d308dae54b0394a364d2988ec6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.365888 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 7c57c5ca55f24621b66197a3698504bd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.366399 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.370050 25470 tablet_service.cc:800] Processing DeleteTablet for tablet d62715e230154e2abe126ef97feec7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.371454 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:55.371930 25623 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.608s (vote granted)
I0619 00:11:55.372148 25623 raft_consensus.cc:2245] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:55.372573 25736 ts_tablet_manager.cc:1382] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:55.372880 25736 log.cc:986] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/397ae10bb6d841b4ba561ec431d29864
I0619 00:11:55.373539 25736 ts_tablet_manager.cc:1403] T 397ae10bb6d841b4ba561ec431d29864 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:55.374615 25069 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC with a delay of 77 ms (attempt = 2)
W0619 00:11:55.374936 25069 catalog_manager.cc:2910] Async tablet task 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:55.375454 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 220) took 2721ms. Request Metrics: {"tablet-delete.queue_time_us":2588591}
I0619 00:11:55.379463 25601 ts_tablet_manager.cc:1173] T 1b15ca9f6e4d4d32bbff1f041a5cb13c P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.376811 25462 ts_tablet_manager.cc:1173] T 455d8e1ee5be484b8005312d2d886821 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.380136 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 775f0b6b309c49278066a5e7cf623af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.382117 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.382491 25464 ts_tablet_manager.cc:1173] T bae6d308dae54b0394a364d2988ec6b3 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.382972 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.383055 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) successfully deleted
I0619 00:11:55.383535 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.383671 25423 leader_election.cc:379] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.383934 25423 leader_election.cc:270] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
W0619 00:11:55.380666 25558 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.384737 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 225) took 2540ms. Request Metrics: {"tablet-delete.queue_time_us":2535431}
I0619 00:11:55.385090 25725 tablet_bootstrap.cc:427] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:55.385304 25736 tablet_replica.cc:261] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:55.385426 25725 ts_tablet_manager.cc:1045] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.138s	user 0.004s	sys 0.001s
I0619 00:11:55.387178 25724 consensus_queue.cc:227] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.387920 25885 raft_consensus.cc:457] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.388133 25885 raft_consensus.cc:2732] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.567s (starting election)
I0619 00:11:55.388519 25885 raft_consensus.cc:479] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.358937 25323 tablet_service.cc:759] Processing CreateTablet for tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c (table=default.table_10c2abdb4e2f45feb095ecf73709e89b [id=191ee261cfbe4ca3b04553067de4e26d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.360695 25325 tablet_service.cc:759] Processing CreateTablet for tablet c2cf56dada3c41d4920a4091a5fdfded (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.389680 25885 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.390326 25071 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 89 ms (attempt = 3)
W0619 00:11:55.390584 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 has no consensus state
I0619 00:11:55.390678 25885 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.392640 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:55.392915 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:55.393146 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:55.393555 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:55.393863 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:55.390365 26008 raft_consensus.cc:457] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.394510 26008 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.709s (starting election)
I0619 00:11:55.394812 25724 ts_tablet_manager.cc:1073] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.059s	user 0.004s	sys 0.000s
I0619 00:11:55.395102 25724 ts_tablet_manager.cc:1028] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:55.395684 25724 tablet_bootstrap.cc:427] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:55.396045 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:55.394822 26008 raft_consensus.cc:479] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.398083 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 300 ms (attempt = 5)
W0619 00:11:55.398352 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:55.398636 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(d62715e230154e2abe126ef97feec7ce)
I0619 00:11:55.399055 25736 raft_consensus.cc:2042] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:55.401221 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 1040 ms (attempt = 7)
W0619 00:11:55.401525 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:55.402164 25724 tablet_bootstrap.cc:578] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:55.402979 25081 catalog_manager.cc:3663] Got report from deleted tablet edfdedbd4eca4b62be24b1649f820d0e (table default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:55.403669 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:11:55.410396 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.411732 25081 catalog_manager.cc:3663] Got report from deleted tablet edfdedbd4eca4b62be24b1649f820d0e (table default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:55.411870 25947 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.170s (election complete)
I0619 00:11:55.412128 25947 raft_consensus.cc:2539] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:55.412204 25736 raft_consensus.cc:2069] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.412358 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.412442 25947 raft_consensus.cc:593] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:55.412914 25947 consensus_queue.cc:204] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.413328 25599 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.413483 25736 ts_tablet_manager.cc:1369] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.412528 25601 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:55.415640 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.415729 25725 raft_consensus.cc:325] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.416329 25725 raft_consensus.cc:351] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.416503 25725 raft_consensus.cc:628] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:55.416960 25725 consensus_queue.cc:227] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.418613 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 397ae10bb6d841b4ba561ec431d29864 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.419342 25079 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 1 got 0
I0619 00:11:55.420081 25079 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 22 ms (attempt = 1)
W0619 00:11:55.420337 25079 catalog_manager.cc:3945] Failed to send 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:55.421202 25725 ts_tablet_manager.cc:1073] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.036s	user 0.004s	sys 0.000s
I0619 00:11:55.421547 25725 ts_tablet_manager.cc:1028] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:55.421916 25725 tablet_bootstrap.cc:427] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:55.376163 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d62715e230154e2abe126ef97feec7ce" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.391367 25330 ts_tablet_manager.cc:1173] T ec8f89d6855544a584eb2ebcd88b7fb6 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.424403 25331 ts_tablet_manager.cc:1173] T eb36a22cf80841b58fd5ce9e6998bf00 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.424052 26008 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.427579 26008 leader_election.cc:243] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.428364 25326 tablet_service.cc:759] Processing CreateTablet for tablet 8e6ecf1d6504414dbb742d841ba8dcde (table=default.table_dc31a7293c29471394747b8090909823 [id=ea28ff41c1f14b7f9a9bffa75122a19a]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:55.430021 25419 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.430418 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.430677 25336 ts_tablet_manager.cc:1173] T 0fee3be144c14edab6db58c99e3cda7d P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.430547 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7277051d52dd4be1a6dfa06488c98293" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:55.431157 25623 raft_consensus.cc:2175] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:55.431645 25331 tablet_service.cc:759] Processing CreateTablet for tablet 2a0b025496ee4087acadc27bbb6d4983 (table=default.table_042bb1895e44487bbf4563658402bf87 [id=8452c371978c48f28ea44cb0fe5293ec]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.433993 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:55.434973 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.435652 25599 tablet_service.cc:759] Processing CreateTablet for tablet 2a0b025496ee4087acadc27bbb6d4983 (table=default.table_042bb1895e44487bbf4563658402bf87 [id=8452c371978c48f28ea44cb0fe5293ec]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.436280 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2a0b025496ee4087acadc27bbb6d4983. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.432432 25290 leader_election.cc:393] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:55.431023 25352 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.437461 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "edfdedbd4eca4b62be24b1649f820d0e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.431100 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:55.438516 25355 raft_consensus.cc:1081] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:11:55.439246 25419 leader_election.cc:306] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.431331 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:55.440306 25419 leader_election.cc:306] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.441701 25601 tablet_service.cc:759] Processing CreateTablet for tablet 8e6ecf1d6504414dbb742d841ba8dcde (table=default.table_dc31a7293c29471394747b8090909823 [id=ea28ff41c1f14b7f9a9bffa75122a19a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.442282 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8e6ecf1d6504414dbb742d841ba8dcde. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.443542 25724 tablet_bootstrap.cc:427] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:55.443694 25796 consensus_queue.cc:963] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:55.444479 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 38 ms (attempt = 2)
I0619 00:11:55.444411 25724 ts_tablet_manager.cc:1045] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.049s	user 0.005s	sys 0.000s
W0619 00:11:55.444777 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:55.441047 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8e6ecf1d6504414dbb742d841ba8dcde. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.445379 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c2cf56dada3c41d4920a4091a5fdfded. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.430048 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 242) took 2655ms. Request Metrics: {"tablet-delete.queue_time_us":2414918}
I0619 00:11:55.447403 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 775f0b6b309c49278066a5e7cf623af2 (table default.table_33483e8524674ab0a5207eb14818073b [id=1960db0181f747909b6f417f161f88ee]) successfully deleted
W0619 00:11:55.438331 25423 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: d62715e230154e2abe126ef97feec7ce
I0619 00:11:55.448417 25423 leader_election.cc:270] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:55.450691 25423 raft_consensus.cc:2442] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:11:55.452970 25069 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC with a delay of 111 ms (attempt = 3)
W0619 00:11:55.453238 25069 catalog_manager.cc:2910] Async tablet task 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:55.454085 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.455653 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2a0b025496ee4087acadc27bbb6d4983. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.456835 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c5a846460af4484aaa4fa8a37934388b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:55.457561 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_53e73552aa6d4161a595d779a9fc96cc" }
W0619 00:11:55.457774 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c5a846460af4484aaa4fa8a37934388b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.457996 25079 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3828fbda3c26454f85595d8e6fbb70e5" }
I0619 00:11:55.457870 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b826e655ec12493da7bd4d816620f96a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.458704 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b826e655ec12493da7bd4d816620f96a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.460394 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_aa2a879352bf4f2d89e870a4cfdc176e"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.461228 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aa2a879352bf4f2d89e870a4cfdc176e in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.462468 25736 ts_tablet_manager.cc:1382] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:11:55.462429 25553 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.462736 25553 leader_election.cc:270] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.462746 25736 log.cc:986] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/d62715e230154e2abe126ef97feec7ce
I0619 00:11:55.462613 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_8efbcf7cfbc0497fb5d8fb885d82e4ca"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:55.463209 25736 ts_tablet_manager.cc:1403] T d62715e230154e2abe126ef97feec7ce P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
W0619 00:11:55.463471 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8efbcf7cfbc0497fb5d8fb885d82e4ca in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.463428 25885 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.590s (election complete)
I0619 00:11:55.463663 25885 raft_consensus.cc:2487] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.452157 25730 ts_tablet_manager.cc:1382] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:55.464442 25730 log.cc:986] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:55.464928 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet 7f60ea5c5def47ee846d0186bbc90c18 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:55.464761 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 231) took 2485ms. Request Metrics: {"mutex_wait_us":12562,"tablet-delete.queue_time_us":2405598}
I0619 00:11:55.465541 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 233) took 2484ms. Request Metrics: {"tablet-delete.queue_time_us":2484001}
I0619 00:11:55.467237 25730 ts_tablet_manager.cc:1403] T a284677c4b7f46e18e25edee6024e3eb P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:11:55.469972 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet d6c3c7f783cb43d59b46502261aeced3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet d6c3c7f783cb43d59b46502261aeced3 already in progress: creating tablet
I0619 00:11:55.470196 25071 catalog_manager.cc:2891] Scheduling retry of d6c3c7f783cb43d59b46502261aeced3 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 61 ms (attempt = 1)
I0619 00:11:55.473980 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 2074 ms (attempt = 8)
W0619 00:11:55.474231 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:55.464915 25724 raft_consensus.cc:325] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.446003 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 243) took 2669ms. Request Metrics: {"tablet-delete.queue_time_us":2669188}
W0619 00:11:55.477056 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 54373917567a479d93dc8a3fd006d2fe with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 54373917567a479d93dc8a3fd006d2fe already in progress: creating tablet
I0619 00:11:55.477293 25067 catalog_manager.cc:2891] Scheduling retry of 54373917567a479d93dc8a3fd006d2fe Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 66 ms (attempt = 2)
I0619 00:11:55.476979 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 244) took 2689ms. Request Metrics: {"tablet-delete.queue_time_us":2642970}
W0619 00:11:55.479709 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a because the tablet was not found. No further retry: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:55.480460 25071 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 131 ms (attempt = 4)
W0619 00:11:55.480772 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 consensus state has no leader
I0619 00:11:55.484055 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 74 ms (attempt = 3)
W0619 00:11:55.484316 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:55.484215 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "edfdedbd4eca4b62be24b1649f820d0e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.485321 25885 raft_consensus.cc:457] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.485559 25885 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.180s (starting election)
I0619 00:11:55.485836 25885 raft_consensus.cc:479] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.486840 25885 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.487221 25885 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:55.490386 25553 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.490303 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:55.491264 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.493438 25323 ts_tablet_manager.cc:1173] T 1b15ca9f6e4d4d32bbff1f041a5cb13c P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:55.494349 25553 leader_election.cc:306] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.496212 25727 tablet_bootstrap.cc:427] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:55.496898 25325 ts_tablet_manager.cc:1173] T c2cf56dada3c41d4920a4091a5fdfded P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.479585 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 246) took 2644ms. Request Metrics: {"tablet-delete.queue_time_us":2643894}
W0619 00:11:55.498312 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 because the tablet was not found. No further retry: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:55.498257 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 245) took 2663ms. Request Metrics: {"tablet-delete.queue_time_us":2657534}
W0619 00:11:55.499058 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet d6c3c7f783cb43d59b46502261aeced3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet d6c3c7f783cb43d59b46502261aeced3 already in progress: creating tablet
I0619 00:11:55.499300 25067 catalog_manager.cc:2891] Scheduling retry of d6c3c7f783cb43d59b46502261aeced3 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 56 ms (attempt = 1)
I0619 00:11:55.500568 25326 ts_tablet_manager.cc:1173] T 8e6ecf1d6504414dbb742d841ba8dcde P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.501981 25598 tablet_service.cc:759] Processing CreateTablet for tablet 48eb7be170a64eb1bd869eba7067843d (table=default.table_5e0e10c382ce4a0b8aeb14cc019630a0 [id=3650722bfdfb4b268310dcdcdd8ea921]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.502568 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 48eb7be170a64eb1bd869eba7067843d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.504717 25323 tablet_service.cc:759] Processing CreateTablet for tablet 4c92d341c4d94e4987524a056e703310 (table=default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.505717 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 544 ms (attempt = 6)
W0619 00:11:55.505990 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:55.506307 25741 tablet_replica.cc:261] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:55.508530 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:55.508805 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:55.509039 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:55.509265 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:55.509517 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:55.509770 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:55.509994 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(a85577b0beba4276b963e5c360544bc7)
I0619 00:11:55.510007 25599 ts_tablet_manager.cc:1173] T 2a0b025496ee4087acadc27bbb6d4983 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.510345 25741 raft_consensus.cc:2042] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:55.511917 25724 raft_consensus.cc:351] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.502343 25727 ts_tablet_manager.cc:1045] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.231s	user 0.006s	sys 0.005s
I0619 00:11:55.512150 25724 raft_consensus.cc:628] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:55.512691 25724 consensus_queue.cc:227] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:55.516289 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.512212 25741 consensus_peers.cc:456] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:55.517276 25741 consensus_peers.cc:456] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
W0619 00:11:55.518332 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.466018 25462 tablet_service.cc:759] Processing CreateTablet for tablet 0fee3be144c14edab6db58c99e3cda7d (table=default.table_c71886d2a0ac4d869a6806cf551bba7e [id=e4d5113658b04af49f2b7d2e67cf71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.520426 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0fee3be144c14edab6db58c99e3cda7d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.486441 25725 tablet_bootstrap.cc:578] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:55.520906 25724 ts_tablet_manager.cc:1073] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.076s	user 0.004s	sys 0.000s
I0619 00:11:55.521694 25741 pending_rounds.cc:63] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Trying to abort 1 pending transactions.
I0619 00:11:55.521719 25724 ts_tablet_manager.cc:1028] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:55.508671 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4c92d341c4d94e4987524a056e703310. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.521845 25741 pending_rounds.cc:68] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390916213264384 op_type: NO_OP noop_request { }
I0619 00:11:55.522184 25724 tablet_bootstrap.cc:427] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:55.522163 25741 raft_consensus.cc:2622] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:55.522377 25741 raft_consensus.cc:2069] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.483633 25487 raft_consensus.cc:2206] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:55.524082 25470 tablet_service.cc:759] Processing CreateTablet for tablet ec8f89d6855544a584eb2ebcd88b7fb6 (table=default.table_77a5f79a6ba04303b379fc9a17e73f9c [id=e81ccf2cf02a4d278717d5fd8274f2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.524669 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ec8f89d6855544a584eb2ebcd88b7fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.530654 25885 raft_consensus.cc:457] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.530906 25885 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.508s (starting election)
I0619 00:11:55.531147 25885 raft_consensus.cc:479] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.532135 25885 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.532519 25885 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.532642 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:55.532578 25290 leader_election.cc:393] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:55.533505 25290 leader_election.cc:270] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.535146 25599 tablet_service.cc:759] Processing CreateTablet for tablet 4c92d341c4d94e4987524a056e703310 (table=default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.535786 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4c92d341c4d94e4987524a056e703310. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.538137 25462 ts_tablet_manager.cc:1173] T 0fee3be144c14edab6db58c99e3cda7d P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.538614 25081 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_6cd711ffd03644e8a98c73f5d95fa03f" } new_table_name: "default.table_297b8bff60264a579f026fbd82396f53" alter_external_catalogs: true
W0619 00:11:55.538861 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.540007 25947 consensus_queue.cc:963] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:11:55.540704 25558 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.540971 25558 leader_election.cc:270] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.541313 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) successfully deleted
I0619 00:11:55.541676 25885 raft_consensus.cc:2732] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.534s (election complete)
I0619 00:11:55.541893 25885 raft_consensus.cc:2487] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.542464 25464 tablet_service.cc:759] Processing CreateTablet for tablet c2cf56dada3c41d4920a4091a5fdfded (table=default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.543066 25087 catalog_manager.cc:3663] Got report from deleted tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:55.543128 25423 leader_election.cc:379] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.543380 25423 leader_election.cc:270] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:55.544208 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 181) took 2146ms. Request Metrics: {"tablet-delete.queue_time_us":1864376}
I0619 00:11:55.544448 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]). Expected version 1 got 0
I0619 00:11:55.544904 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet a284677c4b7f46e18e25edee6024e3eb (table default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]) successfully deleted
I0619 00:11:55.545526 25080 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 21 ms (attempt = 1)
I0619 00:11:55.545460 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 184) took 2078ms. Request Metrics: {"tablet-delete.queue_time_us":2077444}
W0619 00:11:55.545845 25080 catalog_manager.cc:3945] Failed to send 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
W0619 00:11:55.546404 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb because the tablet was not found. No further retry: Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:55.546849 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 186) took 1902ms. Request Metrics: {"tablet-delete.queue_time_us":1896773}
I0619 00:11:55.547708 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 189) took 1790ms. Request Metrics: {"tablet-delete.queue_time_us":1790110}
I0619 00:11:55.547452 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.548418 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 190) took 1791ms. Request Metrics: {"tablet-delete.queue_time_us":1790527}
I0619 00:11:55.549063 25725 tablet_bootstrap.cc:427] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:55.549437 25725 ts_tablet_manager.cc:1045] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.128s	user 0.006s	sys 0.001s
W0619 00:11:55.548219 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a4c653a255984d3bac88df63362b9334 because the tablet was not found. No further retry: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
W0619 00:11:55.550119 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a4c653a255984d3bac88df63362b9334 because the tablet was not found. No further retry: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
W0619 00:11:55.550576 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f75b8a4f1fd845fbab6f83f59b7cfc58 already in progress: creating tablet
I0619 00:11:55.550928 25071 catalog_manager.cc:2891] Scheduling retry of f75b8a4f1fd845fbab6f83f59b7cfc58 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 64 ms (attempt = 1)
I0619 00:11:55.551100 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:55.549733 25598 ts_tablet_manager.cc:1173] T 48eb7be170a64eb1bd869eba7067843d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.538503 25470 ts_tablet_manager.cc:1173] T ec8f89d6855544a584eb2ebcd88b7fb6 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.557824 25079 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet d3c7114381c0423982b9ccbff120c2a2
I0619 00:11:55.557940 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 8ca3a0771b9e41eeb0d14d886f970159
I0619 00:11:55.558145 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d3c7114381c0423982b9ccbff120c2a2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.558262 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.558689 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.559056 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.559437 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d3c7114381c0423982b9ccbff120c2a2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.559825 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d3c7114381c0423982b9ccbff120c2a2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.560196 25331 ts_tablet_manager.cc:1173] T 2a0b025496ee4087acadc27bbb6d4983 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.560470 25068 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 173 ms (attempt = 4)
W0619 00:11:55.560763 25068 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
I0619 00:11:55.561198 25600 tablet_service.cc:800] Processing DeleteTablet for tablet d3c7114381c0423982b9ccbff120c2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.560724 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.561789 25947 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.012s (election complete)
I0619 00:11:55.559146 25083 catalog_manager.cc:3786] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:55.562050 25947 raft_consensus.cc:2539] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:55.562263 25083 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 1 got 0
I0619 00:11:55.562865 25947 raft_consensus.cc:593] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:55.563246 25725 raft_consensus.cc:325] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.565662 25069 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC with a delay of 168 ms (attempt = 4)
W0619 00:11:55.565943 25069 catalog_manager.cc:2910] Async tablet task 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:55.516414 25727 raft_consensus.cc:325] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.566977 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:55.567360 25727 raft_consensus.cc:351] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:55.567389 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:55.567565 25727 raft_consensus.cc:628] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:55.563335 25947 consensus_queue.cc:204] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.568015 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.568625 25068 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 37 ms (attempt = 2)
W0619 00:11:55.568913 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:55.568367 25796 raft_consensus.cc:457] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.569102 25796 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.956s (starting election)
W0619 00:11:55.569355 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.569411 25796 raft_consensus.cc:479] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.569131 25470 tablet_service.cc:759] Processing CreateTablet for tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c (table=default.table_10c2abdb4e2f45feb095ecf73709e89b [id=191ee261cfbe4ca3b04553067de4e26d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.570559 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.570941 25796 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.571393 25796 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.573132 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:55.574537 25419 leader_election.cc:306] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.575605 25601 ts_tablet_manager.cc:1173] T 8e6ecf1d6504414dbb742d841ba8dcde P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.576428 25067 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC with a delay of 1070 ms (attempt = 7)
W0619 00:11:55.576726 25067 catalog_manager.cc:2910] Async tablet task 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 7277051d52dd4be1a6dfa06488c98293 consensus state has no leader
W0619 00:11:55.576872 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.577700 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:55.579018 25553 leader_election.cc:306] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.578500 25724 tablet_bootstrap.cc:578] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:55.579421 25325 tablet_service.cc:759] Processing CreateTablet for tablet 48eb7be170a64eb1bd869eba7067843d (table=default.table_5e0e10c382ce4a0b8aeb14cc019630a0 [id=3650722bfdfb4b268310dcdcdd8ea921]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.580063 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 48eb7be170a64eb1bd869eba7067843d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.582643 25730 tablet_replica.cc:261] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:55.582955 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:55.583212 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:55.583452 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:55.583721 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:55.583760 25081 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 32 ms (attempt = 1)
I0619 00:11:55.583989 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(edfdedbd4eca4b62be24b1649f820d0e)
W0619 00:11:55.584036 25081 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.584200 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:55.584391 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:55.584616 25730 raft_consensus.cc:2042] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:55.584908 25730 raft_consensus.cc:2069] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.586515 25730 ts_tablet_manager.cc:1369] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.589555 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.590554 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c2cf56dada3c41d4920a4091a5fdfded. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.591532 25467 tablet_service.cc:759] Processing CreateTablet for tablet 8e6ecf1d6504414dbb742d841ba8dcde (table=default.table_dc31a7293c29471394747b8090909823 [id=ea28ff41c1f14b7f9a9bffa75122a19a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.592075 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8e6ecf1d6504414dbb742d841ba8dcde. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.593200 25423 leader_election.cc:379] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.593449 25423 leader_election.cc:270] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:55.594384 25468 tablet_service.cc:759] Processing CreateTablet for tablet 2a0b025496ee4087acadc27bbb6d4983 (table=default.table_042bb1895e44487bbf4563658402bf87 [id=8452c371978c48f28ea44cb0fe5293ec]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.594794 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet a4c653a255984d3bac88df63362b9334 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:55.594972 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2a0b025496ee4087acadc27bbb6d4983. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.600127 25464 ts_tablet_manager.cc:1173] T c2cf56dada3c41d4920a4091a5fdfded P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.600435 25736 tablet_replica.cc:261] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:55.602169 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.602473 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.602741 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.603029 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.603371 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.603723 25468 ts_tablet_manager.cc:1173] T 2a0b025496ee4087acadc27bbb6d4983 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.605716 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.606025 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.607868 25068 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 77 ms (attempt = 3)
W0619 00:11:55.608147 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:55.606223 25736 raft_consensus.cc:2042] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:55.608606 25736 consensus_peers.cc:456] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:55.608862 25736 consensus_peers.cc:456] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:55.609516 25736 raft_consensus.cc:2069] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.610594 25725 raft_consensus.cc:351] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:55.592419 25558 leader_election.cc:306] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.610816 25725 raft_consensus.cc:628] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:55.610878 25558 leader_election.cc:270] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.577416 25727 consensus_queue.cc:227] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.611403 25725 consensus_queue.cc:227] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:55.612109 25558 leader_election.cc:306] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.612347 25558 leader_election.cc:270] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.612618 25736 ts_tablet_manager.cc:1369] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.612730 25071 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 260 ms (attempt = 5)
I0619 00:11:55.612892 25885 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.026s (election complete)
I0619 00:11:55.613112 25885 raft_consensus.cc:2487] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:55.613006 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 consensus state has no leader
I0619 00:11:55.613363 25885 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.623s (election complete)
I0619 00:11:55.613562 25885 raft_consensus.cc:2487] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.596788 25724 tablet_bootstrap.cc:427] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:55.616267 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:55.616384 25724 ts_tablet_manager.cc:1045] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.094s	user 0.005s	sys 0.000s
I0619 00:11:55.617121 25940 raft_consensus.cc:2732] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.078s (election complete)
I0619 00:11:55.617362 25940 raft_consensus.cc:2784] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.618892 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 72 ms (attempt = 2)
W0619 00:11:55.619143 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.620903 25724 raft_consensus.cc:325] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:55.616850 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 84292.2us
I0619 00:11:55.627442 25086 catalog_manager.cc:3663] Got report from deleted tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:55.628065 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.628149 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:55.628449 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:55.628906 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.626508 25940 raft_consensus.cc:2487] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.628865 25330 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.629719 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.629465 25326 tablet_service.cc:800] Processing DeleteTablet for tablet d3c7114381c0423982b9ccbff120c2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.629101 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.632304 25086 catalog_manager.cc:3663] Got report from deleted tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:55.632973 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0474be659e5e4a4cbfc181b8c812b1eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:55.634943 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.635218 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 0474be659e5e4a4cbfc181b8c812b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.635910 25727 ts_tablet_manager.cc:1073] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.124s	user 0.002s	sys 0.002s
I0619 00:11:55.636231 25727 ts_tablet_manager.cc:1028] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:55.636564 25727 tablet_bootstrap.cc:427] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:55.639263 25741 ts_tablet_manager.cc:1369] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.640769 25724 raft_consensus.cc:351] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.641469 25724 raft_consensus.cc:628] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:55.641957 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 1032 ms (attempt = 7)
W0619 00:11:55.642220 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:55.643659 25724 consensus_queue.cc:227] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.626801 25601 tablet_service.cc:800] Processing DeleteTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.653344 25724 ts_tablet_manager.cc:1073] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.000s
I0619 00:11:55.655760 25325 ts_tablet_manager.cc:1173] T 48eb7be170a64eb1bd869eba7067843d P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.656762 25083 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_b2764b59c43146a8ae54b361c93b3406" } new_table_name: "default.table_61eaea2a979142bbb875967b756eb1a1" alter_external_catalogs: true
I0619 00:11:55.658208 25323 ts_tablet_manager.cc:1173] T 4c92d341c4d94e4987524a056e703310 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.663537 25727 tablet_bootstrap.cc:578] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:55.666997 25083 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 56 ms (attempt = 1)
W0619 00:11:55.667258 25083 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
W0619 00:11:55.654606 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:55.668859 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.669387 25724 ts_tablet_manager.cc:1028] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:55.669775 25724 tablet_bootstrap.cc:427] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:55.672039 25724 tablet_bootstrap.cc:578] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:55.673938 25069 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 1035 ms (attempt = 7)
W0619 00:11:55.674255 25069 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:55.676290 25599 ts_tablet_manager.cc:1173] T 4c92d341c4d94e4987524a056e703310 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.555126 25462 tablet_service.cc:759] Processing CreateTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 (table=default.table_c5bad852240c4d92bb0cc0cbfccd21c6 [id=fca6fa877b5f4226858ebe70c5d90418]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.613984 25467 ts_tablet_manager.cc:1173] T 8e6ecf1d6504414dbb742d841ba8dcde P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.618840 25847 raft_consensus.cc:457] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.634318 25947 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.239s (election complete)
I0619 00:11:55.641444 25725 ts_tablet_manager.cc:1073] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.092s	user 0.004s	sys 0.000s
I0619 00:11:55.678056 25847 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.743s (starting election)
I0619 00:11:55.678107 25470 ts_tablet_manager.cc:1173] T 1b15ca9f6e4d4d32bbff1f041a5cb13c P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:55.649194 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 56014us
I0619 00:11:55.678352 25847 raft_consensus.cc:479] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.679407 25847 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.679385 25467 tablet_service.cc:759] Processing CreateTablet for tablet 48eb7be170a64eb1bd869eba7067843d (table=default.table_5e0e10c382ce4a0b8aeb14cc019630a0 [id=3650722bfdfb4b268310dcdcdd8ea921]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.679869 25847 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:55.679973 25071 consensus_peers.cc:434] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 -> Peer b2a016cc9f274bc1979068306ec6d15e (127.24.119.61:46325): Couldn't send request to peer b2a016cc9f274bc1979068306ec6d15e for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 14 times.
I0619 00:11:55.680706 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:55.681143 25623 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.407s (vote granted)
I0619 00:11:55.681362 25623 raft_consensus.cc:2245] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:55.685227 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 48eb7be170a64eb1bd869eba7067843d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.685401 25725 ts_tablet_manager.cc:1028] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:55.685797 25725 tablet_bootstrap.cc:427] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:55.685737 25068 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 163 ms (attempt = 4)
W0619 00:11:55.686004 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:55.691576 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 107 ms (attempt = 3)
I0619 00:11:55.691810 25730 ts_tablet_manager.cc:1382] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:11:55.691879 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.692060 25730 log.cc:986] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/edfdedbd4eca4b62be24b1649f820d0e
I0619 00:11:55.678120 25947 raft_consensus.cc:2539] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:55.692510 25730 ts_tablet_manager.cc:1403] T edfdedbd4eca4b62be24b1649f820d0e P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:55.692569 25947 raft_consensus.cc:457] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:55.692786 25947 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.925s (starting election)
I0619 00:11:55.692981 25947 raft_consensus.cc:2784] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.693766 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 1029 ms (attempt = 7)
W0619 00:11:55.694036 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:55.697149 25083 catalog_manager.cc:3663] Got report from deleted tablet a284677c4b7f46e18e25edee6024e3eb (table default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:55.697438 25083 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_ef7e3bdd5d4043abbae5282cce9c6fd7 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:55.697706 25083 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:55.697823 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eb36a22cf80841b58fd5ce9e6998bf00. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.698670 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:55.699015 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:55.699344 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.699385 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 513 ms (attempt = 6)
W0619 00:11:55.699674 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:55.701792 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_91c9b7476b204ab89efb73b7959b877a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.702706 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_91c9b7476b204ab89efb73b7959b877a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.711088 25947 raft_consensus.cc:479] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.712071 25947 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.712442 25947 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.714960 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 1025 ms (attempt = 7)
W0619 00:11:55.715214 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:55.715042 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d0723d646ccd440c850471e54eb7078d"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:55.715740 25467 ts_tablet_manager.cc:1173] T 48eb7be170a64eb1bd869eba7067843d P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:55.715893 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d0723d646ccd440c850471e54eb7078d in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.721838 25462 ts_tablet_manager.cc:1173] T eb36a22cf80841b58fd5ce9e6998bf00 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.724535 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 76 ms (attempt = 2)
W0619 00:11:55.724815 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:11:55.728466 25724 tablet_bootstrap.cc:427] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:55.728847 25724 ts_tablet_manager.cc:1045] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.059s	user 0.007s	sys 0.000s
I0619 00:11:55.730217 25724 raft_consensus.cc:325] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.730871 25724 raft_consensus.cc:351] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.731076 25724 raft_consensus.cc:628] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:55.731637 25724 consensus_queue.cc:227] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.735121 25069 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC with a delay of 266 ms (attempt = 5)
W0619 00:11:55.735388 25069 catalog_manager.cc:2910] Async tablet task 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:55.740025 25083 master_service.cc:523] ReplaceTablet: received request to replace tablet 8e3a356933e749b9a4ade476ec30f475 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:55.741199 25727 tablet_bootstrap.cc:427] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:55.741528 25727 ts_tablet_manager.cc:1045] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.105s	user 0.004s	sys 0.003s
I0619 00:11:55.742795 25727 raft_consensus.cc:325] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.744232 25727 raft_consensus.cc:351] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.744433 25727 raft_consensus.cc:628] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:55.744961 25727 consensus_queue.cc:227] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:55.748373 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:55.749850 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.750398 25727 ts_tablet_manager.cc:1073] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I0619 00:11:55.753089 25741 ts_tablet_manager.cc:1382] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:55.753347 25741 log.cc:986] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a85577b0beba4276b963e5c360544bc7
I0619 00:11:55.753978 25741 ts_tablet_manager.cc:1403] T a85577b0beba4276b963e5c360544bc7 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:55.756587 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]). Expected version 1 got 0
I0619 00:11:55.757319 25080 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 17 ms (attempt = 1)
I0619 00:11:55.757436 25086 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
W0619 00:11:55.757582 25080 catalog_manager.cc:3945] Failed to send 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:55.758095 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:55.758357 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.759021 25325 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.759721 25323 tablet_service.cc:759] Processing CreateTablet for tablet cd5ebc7e2d104e63932e1ef32fa3bf2c (table=default.table_b826e655ec12493da7bd4d816620f96a [id=e0e7b261e7ea4788a75f234e12f4f90e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.760354 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cd5ebc7e2d104e63932e1ef32fa3bf2c. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:55.762303 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.763017 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_45dd93d67cc94f2c8e9e784347d14167" }
I0619 00:11:55.733078 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 191) took 1925ms. Request Metrics: {"tablet-delete.queue_time_us":1745511}
I0619 00:11:55.766441 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 192) took 1940ms. Request Metrics: {"tablet-delete.queue_time_us":1937623}
I0619 00:11:55.767084 25599 tablet_service.cc:759] Processing CreateTablet for tablet cd5ebc7e2d104e63932e1ef32fa3bf2c (table=default.table_b826e655ec12493da7bd4d816620f96a [id=e0e7b261e7ea4788a75f234e12f4f90e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.767525 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 193) took 1905ms. Request Metrics: {"tablet-delete.queue_time_us":1904709}
W0619 00:11:55.738741 25419 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.1:32951, user_credentials={real_user=jenkins}} blocked reactor thread for 88968.9us
I0619 00:11:55.768738 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 194) took 1904ms. Request Metrics: {"tablet-delete.queue_time_us":1903851}
I0619 00:11:55.769279 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet edfdedbd4eca4b62be24b1649f820d0e (table default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]) successfully deleted
I0619 00:11:55.769629 25730 tablet_replica.cc:261] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:55.766942 25086 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_30bcafcc4a6f46e99005fd47447e7dbe" }
I0619 00:11:55.769901 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:55.770189 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:55.770427 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:55.770696 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:55.770563 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a284677c4b7f46e18e25edee6024e3eb" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.771003 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:55.771239 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:55.771461 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:55.771687 25730 raft_consensus.cc:2042] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:55.771986 25730 raft_consensus.cc:2069] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.767735 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cd5ebc7e2d104e63932e1ef32fa3bf2c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.772545 25325 tablet_service.cc:759] Processing CreateTablet for tablet c2e5c2f9fb5c46d4a6790371d6e5bd82 (table=default.table_c5a846460af4484aaa4fa8a37934388b [id=b5d572771ac14542b23884efd6744000]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.773175 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c2e5c2f9fb5c46d4a6790371d6e5bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.775121 25724 ts_tablet_manager.cc:1073] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.046s	user 0.003s	sys 0.001s
I0619 00:11:55.775516 25724 ts_tablet_manager.cc:1028] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:55.775984 25724 tablet_bootstrap.cc:427] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:55.776227 26091 raft_consensus.cc:457] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.776484 26091 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.118s (starting election)
I0619 00:11:55.776757 26091 raft_consensus.cc:479] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.777745 26091 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.777762 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:55.778084 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.778235 26091 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:55.769785 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet d62715e230154e2abe126ef97feec7ce because the tablet was not found. No further retry: Not found: Tablet not found: d62715e230154e2abe126ef97feec7ce
I0619 00:11:55.779878 25462 tablet_service.cc:759] Processing CreateTablet for tablet 4c92d341c4d94e4987524a056e703310 (table=default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.780094 26099 tablet.cc:1200] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
W0619 00:11:55.780315 25419 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.780674 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4c92d341c4d94e4987524a056e703310. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:55.781030 25419 leader_election.cc:306] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.785377 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54adb588955b45b79de9752f254728ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.787756 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:55.788182 25623 raft_consensus.cc:2784] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.788439 25623 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.373s (vote granted)
I0619 00:11:55.788805 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 70 ms (attempt = 2)
W0619 00:11:55.789049 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:55.789472 25467 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:55.789603 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet e9a221a1d9814ef99a0b4ccaa04d1017 already in progress: creating tablet
I0619 00:11:55.789883 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_786d8b2d45904f2da470d39c2cb50010"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.790815 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_786d8b2d45904f2da470d39c2cb50010 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.793900 25725 tablet_bootstrap.cc:578] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:55.789844 25071 catalog_manager.cc:2891] Scheduling retry of e9a221a1d9814ef99a0b4ccaa04d1017 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 51 ms (attempt = 1)
W0619 00:11:55.795094 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 93930ffe285d48d080a8ca4793e871d9 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 93930ffe285d48d080a8ca4793e871d9 already in progress: creating tablet
I0619 00:11:55.795313 25071 catalog_manager.cc:2891] Scheduling retry of 93930ffe285d48d080a8ca4793e871d9 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 63 ms (attempt = 1)
I0619 00:11:55.795300 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c1b38858230546e1a993f452735a5f78"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.796242 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c1b38858230546e1a993f452735a5f78 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.796237 25601 tablet_service.cc:759] Processing CreateTablet for tablet c2e5c2f9fb5c46d4a6790371d6e5bd82 (table=default.table_c5a846460af4484aaa4fa8a37934388b [id=b5d572771ac14542b23884efd6744000]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.796860 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c2e5c2f9fb5c46d4a6790371d6e5bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:55.802165 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 112 ms (attempt = 3)
W0619 00:11:55.802431 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:11:55.810250 25601 ts_tablet_manager.cc:1173] T c2e5c2f9fb5c46d4a6790371d6e5bd82 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.812645 26103 tablet.cc:1200] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:55.814723 25736 ts_tablet_manager.cc:1382] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:55.815259 25079 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_91c9b7476b204ab89efb73b7959b877a" } new_table_name: "default.table_b2e120aa5da945a0b123e7ce5642ac94" alter_external_catalogs: true
I0619 00:11:55.816089 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 153 ms (attempt = 4)
I0619 00:11:55.816367 25323 ts_tablet_manager.cc:1173] T cd5ebc7e2d104e63932e1ef32fa3bf2c P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:55.818559 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.819828 25885 raft_consensus.cc:457] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.820089 25885 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.584s (starting election)
I0619 00:11:55.820340 25885 raft_consensus.cc:479] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.821324 25885 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.821732 25885 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.823482 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 8e3a356933e749b9a4ade476ec30f475
I0619 00:11:55.823819 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
I0619 00:11:55.824802 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
I0619 00:11:55.825470 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
I0619 00:11:55.825938 25083 catalog_manager.cc:4596] ReplaceTablet: tablet 8e3a356933e749b9a4ade476ec30f475 deleted and replaced by tablet beddf1523a854f5b9292c0e4013d7436
I0619 00:11:55.826236 25084 catalog_manager.cc:3663] Got report from deleted tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:55.826803 25084 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]). Expected version 1 got 0
I0619 00:11:55.826762 26104 tablet.cc:1200] T 3722b203a516466489c74ef6013eacae P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:55.828620 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 8e3a356933e749b9a4ade476ec30f475 with delete_type TABLET_DATA_DELETED (replaced by tablet beddf1523a854f5b9292c0e4013d7436) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.828766 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 3e7c4b61f5314e40ad71371155a50b05
I0619 00:11:55.829087 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7c4b61f5314e40ad71371155a50b05 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.829594 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7c4b61f5314e40ad71371155a50b05 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.830005 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7c4b61f5314e40ad71371155a50b05 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.821853 25727 ts_tablet_manager.cc:1028] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:55.830855 25727 tablet_bootstrap.cc:427] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:55.830891 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.832540 25325 ts_tablet_manager.cc:1173] T c2e5c2f9fb5c46d4a6790371d6e5bd82 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.832659 26089 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.832953 26089 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.661s (starting election)
I0619 00:11:55.833204 26089 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.834199 26089 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:55.834928 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 30a9d9fda8d34caeb8242c372439321b
I0619 00:11:55.835032 26089 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.835220 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 30a9d9fda8d34caeb8242c372439321b on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.835598 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 30a9d9fda8d34caeb8242c372439321b on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.835991 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 30a9d9fda8d34caeb8242c372439321b on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.837831 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:55.838485 25623 raft_consensus.cc:2245] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:55.838822 25080 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:55.839613 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:55.839491 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:55.839725 25553 leader_election.cc:306] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:55.840718 25553 leader_election.cc:306] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.840240 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.841357 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.841372 25084 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 47 ms (attempt = 1)
W0619 00:11:55.841714 25084 catalog_manager.cc:3945] Failed to send 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.842479 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 30a9d9fda8d34caeb8242c372439321b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.843410 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.843854 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 30a9d9fda8d34caeb8242c372439321b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.844261 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.846907 25724 tablet_bootstrap.cc:578] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:55.847062 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 247) took 2982ms. Request Metrics: {"tablet-delete.queue_time_us":2633199}
I0619 00:11:55.847878 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet a85577b0beba4276b963e5c360544bc7 (table default.table_d8b3984991a442fe95208294f57d466f [id=91bbf689574642628414fb1fb25efc63]) successfully deleted
I0619 00:11:55.847851 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 248) took 2971ms. Request Metrics: {"tablet-delete.queue_time_us":2970810}
W0619 00:11:55.848469 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:55.848703 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 250) took 2950ms. Request Metrics: {"tablet-delete.queue_time_us":2949397}
W0619 00:11:55.849329 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:55.849802 25068 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 256 ms (attempt = 5)
W0619 00:11:55.850076 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
W0619 00:11:55.850738 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.850450 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 8e3a356933e749b9a4ade476ec30f475 with delete_type TABLET_DATA_DELETED (replaced by tablet beddf1523a854f5b9292c0e4013d7436) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.843152 25599 ts_tablet_manager.cc:1173] T cd5ebc7e2d104e63932e1ef32fa3bf2c P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.850889 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 251) took 2933ms. Request Metrics: {"tablet-delete.queue_time_us":2931307}
W0619 00:11:55.852054 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 because the tablet was not found. No further retry: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:55.852295 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 252) took 2934ms. Request Metrics: {"tablet-delete.queue_time_us":2933564}
W0619 00:11:55.853215 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet d62715e230154e2abe126ef97feec7ce with error code TABLET_NOT_RUNNING: Already present: State transition of tablet d62715e230154e2abe126ef97feec7ce already in progress: creating tablet
I0619 00:11:55.853425 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 45 ms (attempt = 1)
I0619 00:11:55.854538 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54adb588955b45b79de9752f254728ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:55.855007 25623 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.490s (vote granted)
I0619 00:11:55.855237 25623 raft_consensus.cc:2245] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:55.856438 25084 catalog_manager.cc:3663] Got report from deleted tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:55.856894 25741 tablet_replica.cc:261] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:55.857125 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.857079 25730 ts_tablet_manager.cc:1369] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.857185 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.857332 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.857529 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.857743 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.857962 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.858156 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.858340 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(7c57c5ca55f24621b66197a3698504bd)
I0619 00:11:55.858507 25741 raft_consensus.cc:2042] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:55.858783 25741 raft_consensus.cc:2069] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.858988 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.859889 25727 tablet_bootstrap.cc:578] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:55.851928 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.855314 26099 tablet.cc:1200] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:55.864826 25736 log.cc:986] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/7c57c5ca55f24621b66197a3698504bd
I0619 00:11:55.856894 25725 tablet_bootstrap.cc:427] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
W0619 00:11:55.843780 25419 leader_election.cc:306] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.865456 25725 ts_tablet_manager.cc:1045] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.180s	user 0.005s	sys 0.004s
I0619 00:11:55.866930 25725 raft_consensus.cc:325] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.867524 25725 raft_consensus.cc:351] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.867779 25725 raft_consensus.cc:628] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:55.868736 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:11:55.869174 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:55.869592 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 65 ms (attempt = 3)
W0619 00:11:55.869848 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:55.865466 25736 ts_tablet_manager.cc:1403] T 7c57c5ca55f24621b66197a3698504bd P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:55.871703 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.872225 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_ef7e3bdd5d4043abbae5282cce9c6fd7" } new_table_name: "default.table_0666108ed61a422aac531a42773d228d" alter_external_catalogs: true
I0619 00:11:55.874466 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 2090 ms (attempt = 8)
W0619 00:11:55.874761 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_bd5ecdaeb9784fe696613bd457e08878 [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c consensus state has no leader
I0619 00:11:55.875757 25599 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:55.868242 25725 consensus_queue.cc:227] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.877028 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 1058 ms (attempt = 7)
W0619 00:11:55.877286 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
W0619 00:11:55.877595 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.878371 25467 tablet_service.cc:800] Processing DeleteTablet for tablet d3c7114381c0423982b9ccbff120c2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:55.879061 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.882863 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.885681 25087 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_ec6a5ef017d74c7eb1777f1bf058a3da [id=a17737f9d2924cc09f372527fb91188b] alter complete (version 2)
I0619 00:11:55.885867 25079 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 65 ms (attempt = 1)
W0619 00:11:55.886123 25079 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 has no consensus state
I0619 00:11:55.889674 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 236) took 2906ms. Request Metrics: {"tablet-delete.queue_time_us":2554690}
I0619 00:11:55.890307 25736 tablet_replica.cc:261] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:55.890565 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:55.890816 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:55.891041 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:55.891268 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:55.891518 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:55.891757 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 33 ms (attempt = 2)
W0619 00:11:55.892014 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.891780 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:55.892256 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:55.892437 25736 raft_consensus.cc:2042] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:55.891757 25354 raft_consensus.cc:1081] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:55.892719 25736 raft_consensus.cc:2069] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:55.893115 25940 raft_consensus.cc:457] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.893350 25940 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.660s (starting election)
I0619 00:11:55.893590 25940 raft_consensus.cc:479] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.894273 25724 tablet_bootstrap.cc:427] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:55.894546 25940 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.894604 25724 ts_tablet_manager.cc:1045] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.119s	user 0.003s	sys 0.001s
I0619 00:11:55.894949 25940 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.896000 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4c653a255984d3bac88df63362b9334" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.895833 25724 raft_consensus.cc:325] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.896538 26092 consensus_queue.cc:963] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:55.896445 25487 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.713s (vote granted)
I0619 00:11:55.897105 25487 raft_consensus.cc:2245] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
W0619 00:11:55.897409 25290 leader_election.cc:306] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
I0619 00:11:55.897804 25071 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 526 ms (attempt = 6)
W0619 00:11:55.898084 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 consensus state has no leader
I0619 00:11:55.898110 25290 leader_election.cc:379] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.898336 25290 leader_election.cc:270] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:55.898811 25940 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.109s (election complete)
I0619 00:11:55.899027 25940 raft_consensus.cc:2539] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:55.899221 25940 raft_consensus.cc:457] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:55.899411 25940 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.564s (starting election)
I0619 00:11:55.899550 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.899605 25940 raft_consensus.cc:2784] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.900508 25325 tablet_service.cc:800] Processing DeleteTablet for tablet d62715e230154e2abe126ef97feec7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:55.902468 26008 raft_consensus.cc:457] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:55.903138 26008 raft_consensus.cc:2732] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.674s (starting election)
I0619 00:11:55.903378 26008 raft_consensus.cc:479] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.904268 26008 leader_election.cc:243] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.904624 26008 leader_election.cc:243] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.904585 26113 tablet.cc:1200] T 3722b203a516466489c74ef6013eacae P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
W0619 00:11:55.906996 25290 leader_election.cc:306] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
W0619 00:11:55.907413 25290 leader_election.cc:306] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:55.907685 25290 leader_election.cc:270] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.907436 25940 raft_consensus.cc:479] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.879487 25725 ts_tablet_manager.cc:1073] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I0619 00:11:55.908687 26008 raft_consensus.cc:2732] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.928s (election complete)
I0619 00:11:55.908910 26008 raft_consensus.cc:2487] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.908972 25725 ts_tablet_manager.cc:1028] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:55.909365 25725 tablet_bootstrap.cc:427] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:55.909343 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.911463 25725 tablet_bootstrap.cc:578] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:55.912417 25724 raft_consensus.cc:351] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.912639 25724 raft_consensus.cc:628] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:55.913111 25724 consensus_queue.cc:227] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.915544 25724 ts_tablet_manager.cc:1073] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I0619 00:11:55.915894 25724 ts_tablet_manager.cc:1028] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:55.916237 25724 tablet_bootstrap.cc:427] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:55.916970 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 142 ms (attempt = 4)
W0619 00:11:55.917225 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:11:55.918419 25724 tablet_bootstrap.cc:578] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:55.918484 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.919308 26116 tablet.cc:1200] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
W0619 00:11:55.920158 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.922119 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b75c9264a29e43658fcbd7c31b36636a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:55.923013 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b75c9264a29e43658fcbd7c31b36636a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.923208 25736 ts_tablet_manager.cc:1369] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.926132 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 107 ms (attempt = 3)
W0619 00:11:55.926393 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.927781 25467 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:55.928469 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
W0619 00:11:55.930227 25558 leader_election.cc:306] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:55.930496 25558 leader_election.cc:270] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:55.931445 26089 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.562s (election complete)
I0619 00:11:55.931677 26089 raft_consensus.cc:2487] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:55.932448 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:55.934178 25423 leader_election.cc:379] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.934973 25423 leader_election.cc:270] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:55.935459 26100 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.860s (election complete)
I0619 00:11:55.935761 26100 raft_consensus.cc:2539] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:55.935617 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_a9a83994991e4fcf9b36cb9a3d0590b0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:55.936000 26100 raft_consensus.cc:457] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:55.936297 26100 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.986s (starting election)
W0619 00:11:55.936532 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a9a83994991e4fcf9b36cb9a3d0590b0 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:55.936530 26100 raft_consensus.cc:2784] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.936592 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 150 ms (attempt = 4)
W0619 00:11:55.937147 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:55.937729 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet 1dd3f7afe05f49199b18220f9c11ccd2 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:55.942981 26100 raft_consensus.cc:479] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.908529 25940 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.944280 25940 leader_election.cc:243] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:55.944975 25727 tablet_bootstrap.cc:427] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:55.945319 25727 ts_tablet_manager.cc:1045] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.115s	user 0.004s	sys 0.003s
W0619 00:11:55.947006 25290 leader_election.cc:306] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
I0619 00:11:55.950587 25727 raft_consensus.cc:325] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.951298 25727 raft_consensus.cc:351] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:55.952203 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.950167 25741 ts_tablet_manager.cc:1369] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:55.953135 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 37 ms (attempt = 2)
W0619 00:11:55.953424 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 has no consensus state
I0619 00:11:55.955198 25084 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 27 ms (attempt = 1)
W0619 00:11:55.955472 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:55.951509 25727 raft_consensus.cc:628] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:55.956313 25727 consensus_queue.cc:227] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.957921 25727 ts_tablet_manager.cc:1073] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I0619 00:11:55.958215 25727 ts_tablet_manager.cc:1028] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:55.958540 25727 tablet_bootstrap.cc:427] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
W0619 00:11:55.965409 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.966418 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]). Expected version 1 got 0
I0619 00:11:55.966795 25080 catalog_manager.cc:3663] Got report from deleted tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:55.967083 25080 catalog_manager.cc:3663] Got report from deleted tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c): Sending delete request for this tablet
I0619 00:11:55.967774 25725 tablet_bootstrap.cc:427] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:55.968176 25725 ts_tablet_manager.cc:1045] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.059s	user 0.004s	sys 0.000s
I0619 00:11:55.968216 25080 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 58 ms (attempt = 1)
W0619 00:11:55.968499 25080 catalog_manager.cc:3945] Failed to send 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:55.968821 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.969180 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
W0619 00:11:55.969739 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.911494 25462 ts_tablet_manager.cc:1173] T 4c92d341c4d94e4987524a056e703310 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.970110 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:55.970175 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4c653a255984d3bac88df63362b9334" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:55.970657 25490 raft_consensus.cc:2784] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:55.970911 25490 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.564s (vote granted)
I0619 00:11:55.973989 25080 catalog_manager.cc:3663] Got report from deleted tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:55.974723 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:55.975342 25490 raft_consensus.cc:2245] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:55.969856 25725 raft_consensus.cc:325] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.976413 25725 raft_consensus.cc:351] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:55.976495 25290 leader_election.cc:379] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:55.976652 25725 raft_consensus.cc:628] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:55.976774 25290 leader_election.cc:270] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:55.976771 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 296 ms (attempt = 5)
W0619 00:11:55.976810 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 42653.7us
W0619 00:11:55.977020 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.977419 25940 raft_consensus.cc:2732] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.667s (election complete)
I0619 00:11:55.977689 25940 raft_consensus.cc:2539] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:55.977408 25725 consensus_queue.cc:227] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.978170 25940 raft_consensus.cc:593] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:55.980262 25725 ts_tablet_manager.cc:1073] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I0619 00:11:55.980780 25725 ts_tablet_manager.cc:1028] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
W0619 00:11:55.980911 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.978808 25940 consensus_queue.cc:204] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:55.985657 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 78 ms (attempt = 2)
W0619 00:11:55.985927 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:55.986174 25080 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 1 got 0
I0619 00:11:55.987028 25624 raft_consensus.cc:1081] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:55.988631 25079 catalog_manager.cc:3663] Got report from deleted tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:55.989234 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:55.989938 26103 tablet.cc:1200] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:55.991509 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 73 ms (attempt = 3)
W0619 00:11:55.991811 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 consensus state has no leader
W0619 00:11:55.995430 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.997818 25325 tablet_service.cc:759] Processing CreateTablet for tablet 2654c7e3655d46b0bac8251218b538e3 (table=default.table_d0723d646ccd440c850471e54eb7078d [id=f340f028d8264c318e62768c5e6d558a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:55.998421 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2654c7e3655d46b0bac8251218b538e3. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:55.997386 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:55.998760 25724 tablet_bootstrap.cc:427] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:56.000701 25724 ts_tablet_manager.cc:1045] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.085s	user 0.005s	sys 0.001s
I0619 00:11:56.002084 25724 raft_consensus.cc:325] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.002755 25724 raft_consensus.cc:351] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.002797 25069 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC with a delay of 548 ms (attempt = 6)
I0619 00:11:56.002972 25724 raft_consensus.cc:628] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:56.003068 25069 catalog_manager.cc:2910] Async tablet task 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:56.003906 25079 catalog_manager.cc:3663] Got report from deleted tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c): Sending delete request for this tablet
I0619 00:11:56.003479 25724 consensus_queue.cc:227] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.005410 25724 ts_tablet_manager.cc:1073] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I0619 00:11:56.007066 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
I0619 00:11:56.007438 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 1 got 0
W0619 00:11:56.008644 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:56.006728 25599 tablet_service.cc:759] Processing CreateTablet for tablet 2654c7e3655d46b0bac8251218b538e3 (table=default.table_d0723d646ccd440c850471e54eb7078d [id=f340f028d8264c318e62768c5e6d558a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.009202 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2654c7e3655d46b0bac8251218b538e3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.009804 25727 tablet_bootstrap.cc:578] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
W0619 00:11:56.012850 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:56.013898 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.014266 25325 ts_tablet_manager.cc:1173] T 2654c7e3655d46b0bac8251218b538e3 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:55.986464 26099 tablet.cc:1200] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
W0619 00:11:56.016621 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:56.017550 25079 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]). Expected version 3 got 0
I0619 00:11:56.018833 25601 tablet_service.cc:759] Processing CreateTablet for tablet 26455f6f1a47489cab7576d428344fd8 (table=default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.019421 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 26455f6f1a47489cab7576d428344fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.019923 25599 ts_tablet_manager.cc:1173] T 2654c7e3655d46b0bac8251218b538e3 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:56.020051 25419 consensus_peers.cc:434] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 86959e7dda374ba9ac23d3c1fd30d993. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:56.021162 25323 tablet_service.cc:759] Processing CreateTablet for tablet 26455f6f1a47489cab7576d428344fd8 (table=default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.021363 25336 tablet_service.cc:759] Processing CreateTablet for tablet 755b3b7e990a4e4599df319df20b697f (table=default.table_aa2a879352bf4f2d89e870a4cfdc176e [id=7f44780ed587411c89ac42bdcc155046]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.021765 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 26455f6f1a47489cab7576d428344fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.021951 25602 tablet_service.cc:759] Processing CreateTablet for tablet 755b3b7e990a4e4599df319df20b697f (table=default.table_aa2a879352bf4f2d89e870a4cfdc176e [id=7f44780ed587411c89ac42bdcc155046]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.022138 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 755b3b7e990a4e4599df319df20b697f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.022511 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 755b3b7e990a4e4599df319df20b697f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.016206 25725 tablet_bootstrap.cc:427] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:55.999326 25736 ts_tablet_manager.cc:1382] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:56.029004 25736 log.cc:986] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:56.029666 25736 ts_tablet_manager.cc:1403] T 7f60ea5c5def47ee846d0186bbc90c18 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:56.029559 25326 tablet_service.cc:759] Processing CreateTablet for tablet f24f9ab30ad04f39b5ac27e557606ab1 (table=default.table_8efbcf7cfbc0497fb5d8fb885d82e4ca [id=1ab88a99b0be453aa0436685c0307852]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.030134 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f24f9ab30ad04f39b5ac27e557606ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.036026 25724 ts_tablet_manager.cc:1028] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:56.036429 25724 tablet_bootstrap.cc:427] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:56.037854 25067 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 40 ms (attempt = 2)
W0619 00:11:56.038161 25067 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:56.038530 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 138 ms (attempt = 4)
W0619 00:11:56.038820 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:55.991641 25730 ts_tablet_manager.cc:1382] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:56.039183 25730 log.cc:986] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/3537c79c30a1401e8a78b387f08eff58
I0619 00:11:56.040154 25601 ts_tablet_manager.cc:1173] T 26455f6f1a47489cab7576d428344fd8 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.040700 25085 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_951214baec60486e8b27aec35a5923e6 [id=3e2a135384014c7b9626f9a8df3162aa] alter complete (version 1)
I0619 00:11:56.040987 25730 ts_tablet_manager.cc:1403] T 3537c79c30a1401e8a78b387f08eff58 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:55.943996 26100 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.042443 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 195) took 2085ms. Request Metrics: {"tablet-delete.queue_time_us":1812426}
I0619 00:11:56.043414 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 199) took 2034ms. Request Metrics: {"tablet-delete.queue_time_us":2025437}
I0619 00:11:56.046932 25323 ts_tablet_manager.cc:1173] T 26455f6f1a47489cab7576d428344fd8 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.047103 25079 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 44 ms (attempt = 1)
W0619 00:11:56.047482 25079 catalog_manager.cc:3945] Failed to send 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 9d44681acacf4f04b6fcfbd4c9a18e87 consensus state has no leader
I0619 00:11:56.048337 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) successfully deleted
I0619 00:11:56.049170 26100 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.049379 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_b1931959ceee4a6caa4a431f2c8234ba" }
I0619 00:11:56.050846 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 1071 ms (attempt = 7)
W0619 00:11:56.051122 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:56.051196 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:56.052407 25336 ts_tablet_manager.cc:1173] T 755b3b7e990a4e4599df319df20b697f P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.057289 25085 catalog_manager.cc:3663] Got report from deleted tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c): Sending delete request for this tablet
I0619 00:11:56.057663 25085 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:56.058954 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
I0619 00:11:56.059432 25326 ts_tablet_manager.cc:1173] T f24f9ab30ad04f39b5ac27e557606ab1 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.059505 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:56.060879 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 281 ms (attempt = 5)
W0619 00:11:56.061167 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:11:56.050878 25079 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_252d88d7d98b495e80affd5dc6eeb739" }
I0619 00:11:56.065279 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 108 ms (attempt = 3)
W0619 00:11:56.065546 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:56.065650 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 152 ms (attempt = 4)
W0619 00:11:56.065973 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 consensus state has no leader
I0619 00:11:56.031201 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 237) took 3024ms. Request Metrics: {"tablet-delete.queue_time_us":2882812}
I0619 00:11:56.074334 25325 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.076889 25079 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 756057a7c34a4c61a261deaf849e433a
I0619 00:11:56.078055 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 756057a7c34a4c61a261deaf849e433a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.078426 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 756057a7c34a4c61a261deaf849e433a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.078820 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 756057a7c34a4c61a261deaf849e433a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.079635 25741 ts_tablet_manager.cc:1382] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:56.079892 25741 log.cc:986] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/7c57c5ca55f24621b66197a3698504bd
I0619 00:11:56.079778 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 242) took 2859ms. Request Metrics: {"tablet-delete.queue_time_us":2858685}
W0619 00:11:56.078997 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Network error: Failed to ping master at 127.24.119.61:46325: Client connection negotiation failed: client connection to 127.24.119.61:46325: connect: Connection refused (error 111)
I0619 00:11:56.080127 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.081246 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.083829 25079 master_service.cc:523] ReplaceTablet: received request to replace tablet 6040e369e0fb445fb993d12dd883039b from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:56.083973 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) successfully deleted
I0619 00:11:56.076571 25725 tablet_bootstrap.cc:578] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:56.084632 25419 leader_election.cc:306] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.046649 25602 ts_tablet_manager.cc:1173] T 755b3b7e990a4e4599df319df20b697f P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.038698 25724 tablet_bootstrap.cc:578] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:56.082672 25558 leader_election.cc:306] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:56.086621 25290 consensus_peers.cc:434] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 7f60ea5c5def47ee846d0186bbc90c18. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:56.086963 25558 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.087857 26089 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.055s (election complete)
I0619 00:11:56.088073 26089 raft_consensus.cc:2487] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.084911 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 756057a7c34a4c61a261deaf849e433a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.090215 25730 tablet_replica.cc:261] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:56.084689 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 248) took 2732ms. Request Metrics: {"tablet-delete.queue_time_us":2710428}
I0619 00:11:56.092427 25069 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 38 ms (attempt = 2)
I0619 00:11:56.090921 25624 raft_consensus.cc:1081] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:11:56.092711 25069 catalog_manager.cc:2910] Async tablet task 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d44681acacf4f04b6fcfbd4c9a18e87 consensus state has no leader
I0619 00:11:56.093171 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 256 ms (attempt = 5)
I0619 00:11:56.092921 25727 tablet_bootstrap.cc:427] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
W0619 00:11:56.093430 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:56.093632 25727 ts_tablet_manager.cc:1045] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.135s	user 0.004s	sys 0.003s
I0619 00:11:56.093793 25741 ts_tablet_manager.cc:1403] T 7c57c5ca55f24621b66197a3698504bd P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:56.094023 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 1dd3f7afe05f49199b18220f9c11ccd2 already in progress: creating tablet
I0619 00:11:56.094246 25071 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 41 ms (attempt = 1)
I0619 00:11:56.094184 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 249) took 2642ms. Request Metrics: {"tablet-delete.queue_time_us":2630848}
I0619 00:11:56.094847 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 250) took 2620ms. Request Metrics: {"tablet-delete.queue_time_us":2619848}
I0619 00:11:56.094919 25601 tablet_service.cc:759] Processing CreateTablet for tablet f24f9ab30ad04f39b5ac27e557606ab1 (table=default.table_8efbcf7cfbc0497fb5d8fb885d82e4ca [id=1ab88a99b0be453aa0436685c0307852]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.095393 25736 tablet_replica.cc:261] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:56.095032 25727 raft_consensus.cc:325] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.095515 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f24f9ab30ad04f39b5ac27e557606ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.095631 25727 raft_consensus.cc:351] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.095641 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:56.095819 25727 raft_consensus.cc:628] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:56.095870 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:56.096067 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:56.093722 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:56.097483 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:56.097982 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:56.098518 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:56.099329 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 254) took 3151ms. Request Metrics: {"tablet-delete.queue_time_us":2908641}
I0619 00:11:56.099722 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:56.099973 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:56.100177 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:56.100353 25730 raft_consensus.cc:2042] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.100615 25730 raft_consensus.cc:2069] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.101269 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:56.101737 25624 raft_consensus.cc:2784] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.101979 25624 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.239s (vote granted)
I0619 00:11:56.105185 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 17e165e0995a4c029aa31392b9bbc04a
I0619 00:11:56.105554 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.105980 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.106386 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.107201 25624 raft_consensus.cc:2245] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:56.109289 25724 tablet_bootstrap.cc:427] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:56.109622 25724 ts_tablet_manager.cc:1045] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.073s	user 0.005s	sys 0.000s
I0619 00:11:56.109800 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) successfully deleted
I0619 00:11:56.106268 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 255) took 3097ms. Request Metrics: {"tablet-delete.queue_time_us":3096434}
I0619 00:11:56.110182 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ff6a2c70c1d94d7a814d423d400f6c33"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:56.110569 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_77a5f79a6ba04303b379fc9a17e73f9c" } new_table_name: "default.table_462e0451fdd448aab866e586f244c58f" alter_external_catalogs: true
I0619 00:11:56.110424 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_4f9bc225373c46aaa2367ff751ca3698"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:56.111093 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ff6a2c70c1d94d7a814d423d400f6c33 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:56.111271 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4f9bc225373c46aaa2367ff751ca3698 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.111831 25068 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 547 ms (attempt = 6)
I0619 00:11:56.096244 25727 consensus_queue.cc:227] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:56.112097 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:56.111204 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 256) took 3099ms. Request Metrics: {"tablet-delete.queue_time_us":3098951}
I0619 00:11:56.112736 25748 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.61:46325
I0619 00:11:56.113245 25087 catalog_manager.cc:3663] Got report from deleted tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:56.113160 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 262) took 2870ms. Request Metrics: {"tablet-delete.queue_time_us":2869508}
I0619 00:11:56.113713 25741 tablet_replica.cc:261] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:56.113899 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a33940e5347489993ab4679e9aeafde on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:56.113940 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:56.114145 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:56.114370 25748 webserver.cc:175] Starting webserver on 127.24.119.61:33317
I0619 00:11:56.114542 25748 webserver.cc:186] Document root disabled
I0619 00:11:56.114338 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:56.115394 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:56.115686 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:56.115916 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:56.120075 25067 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 67 ms (attempt = 3)
W0619 00:11:56.120368 25067 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:56.120682 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 7c57c5ca55f24621b66197a3698504bd (table default.table_aa87e3f6cd96435ab36cdbbc83a51374 [id=68b201dd31454f02838aaa70b04c3c31]) successfully deleted
W0619 00:11:56.121816 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet c18355df78d344839a56189c7549ab1f because the tablet was not found. No further retry: Not found: Tablet not found: c18355df78d344839a56189c7549ab1f
I0619 00:11:56.109916 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.126237 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.131425 25069 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 83 ms (attempt = 3)
W0619 00:11:56.131719 25069 catalog_manager.cc:2910] Async tablet task 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d44681acacf4f04b6fcfbd4c9a18e87 consensus state has no leader
W0619 00:11:56.133009 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet c18355df78d344839a56189c7549ab1f because the tablet was not found. No further retry: Not found: Tablet not found: c18355df78d344839a56189c7549ab1f
W0619 00:11:56.133481 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 6040e369e0fb445fb993d12dd883039b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6040e369e0fb445fb993d12dd883039b already in progress: creating tablet
I0619 00:11:56.133710 25067 catalog_manager.cc:2891] Scheduling retry of 6040e369e0fb445fb993d12dd883039b Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 37 ms (attempt = 1)
I0619 00:11:56.133940 25087 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:56.134248 25087 catalog_manager.cc:3663] Got report from deleted tablet d62715e230154e2abe126ef97feec7ce (table default.table_cc45383880634c41948247ee9fa098d1 [id=fa287205f19a461fa0ba005f9991513b]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:56.101815 25467 tablet_service.cc:759] Processing CreateTablet for tablet cd5ebc7e2d104e63932e1ef32fa3bf2c (table=default.table_b826e655ec12493da7bd4d816620f96a [id=e0e7b261e7ea4788a75f234e12f4f90e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.101985 25725 tablet_bootstrap.cc:427] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:56.103371 25462 tablet_service.cc:759] Processing CreateTablet for tablet c2e5c2f9fb5c46d4a6790371d6e5bd82 (table=default.table_c5a846460af4484aaa4fa8a37934388b [id=b5d572771ac14542b23884efd6744000]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.135040 25725 ts_tablet_manager.cc:1045] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.119s	user 0.003s	sys 0.003s
I0619 00:11:56.135124 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:56.135478 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:56.136481 25725 raft_consensus.cc:325] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.137120 25725 raft_consensus.cc:351] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.135078 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cd5ebc7e2d104e63932e1ef32fa3bf2c. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:56.116211 25423 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:56.137442 25423 leader_election.cc:270] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.137966 25085 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 21 ms (attempt = 1)
I0619 00:11:56.138404 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 8e3a356933e749b9a4ade476ec30f475 with delete_type TABLET_DATA_DELETED (replaced by tablet beddf1523a854f5b9292c0e4013d7436) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:56.138744 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:56.139118 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.140733 26091 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.135s (election complete)
I0619 00:11:56.140995 26091 raft_consensus.cc:2487] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.141844 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 30a9d9fda8d34caeb8242c372439321b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.142211 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.137308 25725 raft_consensus.cc:628] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:56.142671 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:56.143057 25725 consensus_queue.cc:227] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.144619 25725 ts_tablet_manager.cc:1073] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I0619 00:11:56.144971 25725 ts_tablet_manager.cc:1028] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:56.145314 25725 tablet_bootstrap.cc:427] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:56.147346 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 6040e369e0fb445fb993d12dd883039b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 6040e369e0fb445fb993d12dd883039b already in progress: creating tablet
I0619 00:11:56.145332 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c2e5c2f9fb5c46d4a6790371d6e5bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.148636 25423 leader_election.cc:379] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.148921 25423 leader_election.cc:270] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:56.148536 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.149368 26091 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.064s (election complete)
I0619 00:11:56.149633 26091 raft_consensus.cc:2539] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:56.150336 26091 raft_consensus.cc:593] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:56.150754 26091 consensus_queue.cc:204] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.152772 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:56.153092 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:56.153343 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:56.152772 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.153610 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(a284677c4b7f46e18e25edee6024e3eb)
I0619 00:11:56.153856 25736 raft_consensus.cc:2042] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.154122 25736 raft_consensus.cc:2069] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.155275 25736 ts_tablet_manager.cc:1369] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.118726 25727 ts_tablet_manager.cc:1073] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.000s
I0619 00:11:56.122563 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 756057a7c34a4c61a261deaf849e433a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.124754 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.157065 25727 ts_tablet_manager.cc:1028] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:56.157006 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(397ae10bb6d841b4ba561ec431d29864)
I0619 00:11:56.157384 25741 raft_consensus.cc:2042] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.157433 25727 tablet_bootstrap.cc:427] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:56.157692 25741 raft_consensus.cc:2069] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.158069 25730 ts_tablet_manager.cc:1369] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.149260 25071 catalog_manager.cc:2891] Scheduling retry of 6040e369e0fb445fb993d12dd883039b Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 40 ms (attempt = 1)
I0619 00:11:56.160107 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.162204 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 0a33940e5347489993ab4679e9aeafde with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:56.160924 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:56.164626 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.165089 25325 tablet_service.cc:800] Processing DeleteTablet for tablet d62715e230154e2abe126ef97feec7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:56.165648 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:56.166229 25467 ts_tablet_manager.cc:1173] T cd5ebc7e2d104e63932e1ef32fa3bf2c P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.169121 25741 ts_tablet_manager.cc:1369] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.169205 25748 webserver.cc:313] Webserver started. Bound to: http://127.24.119.61:33317/
I0619 00:11:56.163599 25423 leader_election.cc:379] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.170167 25423 leader_election.cc:270] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:56.167156 25080 catalog_manager.cc:3663] Got report from deleted tablet 54373917567a479d93dc8a3fd006d2fe (table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:56.125666 25724 raft_consensus.cc:325] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.171062 25724 raft_consensus.cc:351] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.171159 26100 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.802s (election complete)
I0619 00:11:56.171281 25724 raft_consensus.cc:628] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:56.171401 26100 raft_consensus.cc:2539] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:56.171625 26100 raft_consensus.cc:457] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:56.171823 26100 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.633s (starting election)
I0619 00:11:56.170485 25727 tablet_bootstrap.cc:578] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:56.172019 26100 raft_consensus.cc:2784] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.171916 25724 consensus_queue.cc:227] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.174691 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 147 ms (attempt = 4)
I0619 00:11:56.174790 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
W0619 00:11:56.174973 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:56.176066 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.176596 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:56.176968 25079 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:56.178074 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:56.178720 25423 consensus_peers.cc:434] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 3537c79c30a1401e8a78b387f08eff58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:56.179561 25470 tablet_service.cc:759] Processing CreateTablet for tablet 2654c7e3655d46b0bac8251218b538e3 (table=default.table_d0723d646ccd440c850471e54eb7078d [id=f340f028d8264c318e62768c5e6d558a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.179882 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 302 ms (attempt = 5)
I0619 00:11:56.179483 26100 raft_consensus.cc:479] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.180182 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2654c7e3655d46b0bac8251218b538e3. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:56.180135 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:56.180609 26100 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.181288 25462 ts_tablet_manager.cc:1173] T c2e5c2f9fb5c46d4a6790371d6e5bd82 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.182992 25947 raft_consensus.cc:457] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.183236 25947 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.732s (starting election)
I0619 00:11:56.183461 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54adb588955b45b79de9752f254728ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:56.183471 25947 raft_consensus.cc:479] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.184389 25947 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.184774 25947 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:56.185142 25419 leader_election.cc:306] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.185837 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 49 ms (attempt = 2)
W0619 00:11:56.186079 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:56.186941 25724 ts_tablet_manager.cc:1073] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.064s	user 0.004s	sys 0.000s
I0619 00:11:56.187274 25724 ts_tablet_manager.cc:1028] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:56.187635 25724 tablet_bootstrap.cc:427] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:56.187575 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "981d862d1bab4b14bb10e6fc2174e1cc" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:56.188923 25419 leader_election.cc:306] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.189330 26100 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.192010 25725 tablet_bootstrap.cc:578] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:56.198489 25467 tablet_service.cc:759] Processing CreateTablet for tablet 26455f6f1a47489cab7576d428344fd8 (table=default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.199080 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.201721 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "981d862d1bab4b14bb10e6fc2174e1cc" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:56.202796 25067 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 172 ms (attempt = 4)
I0619 00:11:56.203347 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 4099 ms (attempt = 9)
W0619 00:11:56.203521 25067 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
W0619 00:11:56.204361 25290 consensus_peers.cc:434] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet 7f60ea5c5def47ee846d0186bbc90c18. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:56.204246 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54adb588955b45b79de9752f254728ea" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
W0619 00:11:56.203626 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:56.204661 25623 raft_consensus.cc:2784] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.204923 25623 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.990s (vote granted)
I0619 00:11:56.205847 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c5bad852240c4d92bb0cc0cbfccd21c6" } new_table_name: "default.table_c553f0db13024123bd9fe8fb7de37a72" alter_external_catalogs: true
I0619 00:11:56.209455 25623 raft_consensus.cc:2245] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:56.210122 25087 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:56.210767 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.214339 25725 tablet_bootstrap.cc:427] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:56.215102 25069 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 158 ms (attempt = 4)
W0619 00:11:56.215363 25069 catalog_manager.cc:2910] Async tablet task 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d44681acacf4f04b6fcfbd4c9a18e87 consensus state has no leader
I0619 00:11:56.216338 25601 ts_tablet_manager.cc:1173] T f24f9ab30ad04f39b5ac27e557606ab1 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.218361 25725 ts_tablet_manager.cc:1045] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.073s	user 0.005s	sys 0.000s
I0619 00:11:56.217767 25085 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 (table default.table_c553f0db13024123bd9fe8fb7de37a72 [id=fca6fa877b5f4226858ebe70c5d90418]) Alter Table RPC with a delay of 45 ms (attempt = 1)
W0619 00:11:56.218571 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb36a22cf80841b58fd5ce9e6998bf00 consensus state has no leader
I0619 00:11:56.218945 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 1060 ms (attempt = 7)
W0619 00:11:56.219243 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:56.219638 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 276 ms (attempt = 5)
W0619 00:11:56.219976 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 consensus state has no leader
I0619 00:11:56.219887 25725 raft_consensus.cc:325] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.221413 25725 raft_consensus.cc:351] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.221602 25725 raft_consensus.cc:628] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:56.215900 25470 ts_tablet_manager.cc:1173] T 2654c7e3655d46b0bac8251218b538e3 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.223001 25725 consensus_queue.cc:227] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.224156 25071 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 4140 ms (attempt = 9)
W0619 00:11:56.224437 25071 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:56.224963 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.227121 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c053afda87e949b588e952600c8a012c"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:56.228317 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c053afda87e949b588e952600c8a012c in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.223999 25468 tablet_service.cc:759] Processing CreateTablet for tablet 755b3b7e990a4e4599df319df20b697f (table=default.table_aa2a879352bf4f2d89e870a4cfdc176e [id=7f44780ed587411c89ac42bdcc155046]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.225896 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.225752 26100 consensus_queue.cc:963] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:56.226884 25423 leader_election.cc:379] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.227188 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 756057a7c34a4c61a261deaf849e433a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.227720 25470 tablet_service.cc:759] Processing CreateTablet for tablet f24f9ab30ad04f39b5ac27e557606ab1 (table=default.table_8efbcf7cfbc0497fb5d8fb885d82e4ca [id=1ab88a99b0be453aa0436685c0307852]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.229784 26091 consensus_queue.cc:963] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:56.230438 25423 leader_election.cc:270] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:56.231209 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 26455f6f1a47489cab7576d428344fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.232005 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.232076 26091 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.885s (election complete)
I0619 00:11:56.232301 26091 raft_consensus.cc:2539] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:56.232570 26091 raft_consensus.cc:593] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:56.232772 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_9fa9d2f02d024d7592c83d283fbdff04"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:56.233002 26091 consensus_queue.cc:204] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:56.233660 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9fa9d2f02d024d7592c83d283fbdff04 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.235509 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 109 ms (attempt = 3)
W0619 00:11:56.235782 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:56.236464 25727 tablet_bootstrap.cc:427] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:56.236853 25727 ts_tablet_manager.cc:1045] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.080s	user 0.005s	sys 0.000s
I0619 00:11:56.238842 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 755b3b7e990a4e4599df319df20b697f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.238170 25727 raft_consensus.cc:325] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.240236 25727 raft_consensus.cc:351] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.240425 25727 raft_consensus.cc:628] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:56.240947 25727 consensus_queue.cc:227] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.241727 26008 raft_consensus.cc:457] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.241940 26008 raft_consensus.cc:2732] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 3.185s (starting election)
I0619 00:11:56.242180 26008 raft_consensus.cc:479] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.243090 26008 leader_election.cc:243] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.243460 26008 leader_election.cc:243] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.246521 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:56.246870 25290 leader_election.cc:306] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:56.247493 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet a284677c4b7f46e18e25edee6024e3eb from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:56.248316 25084 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:56.248867 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f24f9ab30ad04f39b5ac27e557606ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.248971 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:56.250416 25468 ts_tablet_manager.cc:1173] T 755b3b7e990a4e4599df319df20b697f P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:56.251050 25290 leader_election.cc:306] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:56.251281 25290 leader_election.cc:270] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.251353 25727 ts_tablet_manager.cc:1073] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I0619 00:11:56.251756 26008 raft_consensus.cc:2732] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.694s (election complete)
I0619 00:11:56.251968 26008 raft_consensus.cc:2487] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.252326 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.252622 25727 ts_tablet_manager.cc:1028] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:56.252987 25727 tablet_bootstrap.cc:427] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:56.254026 25084 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:56.254671 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:56.258776 25724 tablet_bootstrap.cc:578] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:56.259811 25725 ts_tablet_manager.cc:1073] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.041s	user 0.004s	sys 0.000s
I0619 00:11:56.260583 25725 ts_tablet_manager.cc:1028] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:56.256108 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.261761 25727 tablet_bootstrap.cc:578] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:56.262326 25725 tablet_bootstrap.cc:427] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:56.264796 25069 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 (table default.table_c553f0db13024123bd9fe8fb7de37a72 [id=fca6fa877b5f4226858ebe70c5d90418]) Alter Table RPC with a delay of 40 ms (attempt = 2)
W0619 00:11:56.265108 25069 catalog_manager.cc:2910] Async tablet task eb36a22cf80841b58fd5ce9e6998bf00 (table default.table_c553f0db13024123bd9fe8fb7de37a72 [id=fca6fa877b5f4226858ebe70c5d90418]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet eb36a22cf80841b58fd5ce9e6998bf00 consensus state has no leader
I0619 00:11:56.272871 25470 ts_tablet_manager.cc:1173] T f24f9ab30ad04f39b5ac27e557606ab1 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.273524 25748 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/master-1/data/info.pb
I0619 00:11:56.275872 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 514 ms (attempt = 6)
I0619 00:11:56.275203 25080 catalog_manager.cc:3786] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
W0619 00:11:56.276137 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:56.277797 25325 tablet_service.cc:759] Processing CreateTablet for tablet 2a49647c4d13452eac7a2747e4c62040 (table=default.table_b75c9264a29e43658fcbd7c31b36636a [id=b1ae4aee174a4d2b9cdf02b69bf3a8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.280180 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 25748
I0619 00:11:56.282323 25326 tablet_service.cc:759] Processing CreateTablet for tablet beddf1523a854f5b9292c0e4013d7436 (table=default.table_a822c7bace2a4cfd9c1c7f0a9c992551 [id=0d90b232db8e48ed9bfab0c59c6fbdba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.283753 25336 tablet_service.cc:759] Processing CreateTablet for tablet 0c1f35568fe24c6c8cd2b88871b4520d (table=default.table_a9a83994991e4fcf9b36cb9a3d0590b0 [id=1bb609c5a8114109b7cfcfd006a2143f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.284488 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet beddf1523a854f5b9292c0e4013d7436. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.284927 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0c1f35568fe24c6c8cd2b88871b4520d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.285192 26089 raft_consensus.cc:457] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.284622 25741 ts_tablet_manager.cc:1382] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:56.285445 26089 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.620s (starting election)
I0619 00:11:56.285498 25741 log.cc:986] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/397ae10bb6d841b4ba561ec431d29864
I0619 00:11:56.285698 26089 raft_consensus.cc:479] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.286659 26089 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.287046 26089 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.288175 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2a49647c4d13452eac7a2747e4c62040. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.288206 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.289010 26008 raft_consensus.cc:457] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.289250 26008 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.784s (starting election)
W0619 00:11:56.289513 25553 leader_election.cc:306] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.289489 26008 raft_consensus.cc:479] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.290125 25741 ts_tablet_manager.cc:1403] T 397ae10bb6d841b4ba561ec431d29864 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:56.290431 26008 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.290853 26008 leader_election.cc:243] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.294718 25730 ts_tablet_manager.cc:1382] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:56.294944 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:56.295486 25623 raft_consensus.cc:2175] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:56.295424 25336 ts_tablet_manager.cc:1173] T 0c1f35568fe24c6c8cd2b88871b4520d P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.296232 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 263) took 3018ms. Request Metrics: {"tablet-delete.queue_time_us":2790063}
I0619 00:11:56.296492 25290 leader_election.cc:393] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:56.297030 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 265) took 3008ms. Request Metrics: {"tablet-delete.queue_time_us":2971372}
I0619 00:11:56.297379 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 397ae10bb6d841b4ba561ec431d29864 (table default.table_3d4be1da5dd54e91b7a565ceded2dae3 [id=12219e0bf4434aa398adb8604bc8ae14]) successfully deleted
W0619 00:11:56.297876 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a284677c4b7f46e18e25edee6024e3eb with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a284677c4b7f46e18e25edee6024e3eb already in progress: creating tablet
I0619 00:11:56.297893 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 266) took 2839ms. Request Metrics: {"tablet-delete.queue_time_us":2837533}
I0619 00:11:56.298097 25067 catalog_manager.cc:2891] Scheduling retry of a284677c4b7f46e18e25edee6024e3eb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 24 ms (attempt = 1)
W0619 00:11:56.298499 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:56.294961 25730 log.cc:986] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:56.298820 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 271) took 2552ms. Request Metrics: {"tablet-delete.queue_time_us":2551694}
W0619 00:11:56.299425 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f75b8a4f1fd845fbab6f83f59b7cfc58 already in progress: creating tablet
I0619 00:11:56.299628 25067 catalog_manager.cc:2891] Scheduling retry of f75b8a4f1fd845fbab6f83f59b7cfc58 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 52 ms (attempt = 1)
I0619 00:11:56.300020 25730 ts_tablet_manager.cc:1403] T 981d862d1bab4b14bb10e6fc2174e1cc P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:56.300282 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c553f0db13024123bd9fe8fb7de37a72" }
I0619 00:11:56.301435 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 198) took 2306ms. Request Metrics: {"tablet-delete.queue_time_us":2072094}
I0619 00:11:56.302865 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) successfully deleted
I0619 00:11:56.303879 25601 tablet_service.cc:759] Processing CreateTablet for tablet 2a49647c4d13452eac7a2747e4c62040 (table=default.table_b75c9264a29e43658fcbd7c31b36636a [id=b1ae4aee174a4d2b9cdf02b69bf3a8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.304059 25326 ts_tablet_manager.cc:1173] T beddf1523a854f5b9292c0e4013d7436 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.304407 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2a49647c4d13452eac7a2747e4c62040. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.302311 25736 ts_tablet_manager.cc:1382] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:56.305244 25736 log.cc:986] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:56.305794 25069 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 (table default.table_c553f0db13024123bd9fe8fb7de37a72 [id=fca6fa877b5f4226858ebe70c5d90418]) Alter Table RPC with a delay of 112 ms (attempt = 3)
W0619 00:11:56.306056 25069 catalog_manager.cc:2910] Async tablet task eb36a22cf80841b58fd5ce9e6998bf00 (table default.table_c553f0db13024123bd9fe8fb7de37a72 [id=fca6fa877b5f4226858ebe70c5d90418]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet eb36a22cf80841b58fd5ce9e6998bf00 consensus state has no leader
I0619 00:11:56.308578 25725 tablet_bootstrap.cc:578] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:56.310799 26089 raft_consensus.cc:457] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.311048 26089 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.695s (starting election)
I0619 00:11:56.311292 26089 raft_consensus.cc:479] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.312290 26089 leader_election.cc:243] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.312713 26089 leader_election.cc:243] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.314519 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e5e691e6d374cd4bba1f18d8c821723" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.315407 25599 tablet_service.cc:759] Processing CreateTablet for tablet beddf1523a854f5b9292c0e4013d7436 (table=default.table_a822c7bace2a4cfd9c1c7f0a9c992551 [id=0d90b232db8e48ed9bfab0c59c6fbdba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.316037 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet beddf1523a854f5b9292c0e4013d7436. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:56.316388 25553 leader_election.cc:306] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.316964 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_2e9154601fc34c21a4a6c3e101a644dc" }
I0619 00:11:56.319000 25602 tablet_service.cc:759] Processing CreateTablet for tablet 0c1f35568fe24c6c8cd2b88871b4520d (table=default.table_a9a83994991e4fcf9b36cb9a3d0590b0 [id=1bb609c5a8114109b7cfcfd006a2143f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.319536 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0c1f35568fe24c6c8cd2b88871b4520d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.320513 25727 tablet_bootstrap.cc:427] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:56.320864 25727 ts_tablet_manager.cc:1045] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.068s	user 0.005s	sys 0.000s
I0619 00:11:56.323619 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 261 ms (attempt = 5)
W0619 00:11:56.323918 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:56.328529 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.331487 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet eb36a22cf80841b58fd5ce9e6998bf00
I0619 00:11:56.331797 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet eb36a22cf80841b58fd5ce9e6998bf00 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.332175 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet eb36a22cf80841b58fd5ce9e6998bf00 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.332542 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet eb36a22cf80841b58fd5ce9e6998bf00 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.333076 25326 tablet_service.cc:800] Processing DeleteTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.334472 25600 tablet_service.cc:800] Processing DeleteTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.337059 25599 ts_tablet_manager.cc:1173] T beddf1523a854f5b9292c0e4013d7436 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.337841 25602 ts_tablet_manager.cc:1173] T 0c1f35568fe24c6c8cd2b88871b4520d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.338397 25336 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.340430 25085 master_service.cc:523] ReplaceTablet: received request to replace tablet 0c1f35568fe24c6c8cd2b88871b4520d from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:56.342586 25724 tablet_bootstrap.cc:427] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:56.342666 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 3e7724f967ae41d7b523055d1a4f58db
I0619 00:11:56.342967 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.343044 25724 ts_tablet_manager.cc:1045] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.156s	user 0.006s	sys 0.000s
I0619 00:11:56.343334 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.343466 25325 ts_tablet_manager.cc:1173] T 2a49647c4d13452eac7a2747e4c62040 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.343724 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.344825 25724 raft_consensus.cc:325] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.345769 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 525 ms (attempt = 6)
W0619 00:11:56.346057 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:11:56.346420 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 172 ms (attempt = 4)
W0619 00:11:56.346712 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:56.347321 25724 raft_consensus.cc:351] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.347549 25724 raft_consensus.cc:628] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:56.348147 25724 consensus_queue.cc:227] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.350984 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 542 ms (attempt = 6)
W0619 00:11:56.351261 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:56.352124 25079 catalog_manager.cc:3663] Got report from deleted tablet d3c7114381c0423982b9ccbff120c2a2 (table default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:56.352829 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d3c7114381c0423982b9ccbff120c2a2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:56.355252 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.356011 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.357331 25599 tablet_service.cc:800] Processing DeleteTablet for tablet d3c7114381c0423982b9ccbff120c2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.357549 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 0c1f35568fe24c6c8cd2b88871b4520d
I0619 00:11:56.357872 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0c1f35568fe24c6c8cd2b88871b4520d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 8140397d408e47129a182fb6255a0a39)
I0619 00:11:56.358252 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0c1f35568fe24c6c8cd2b88871b4520d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 8140397d408e47129a182fb6255a0a39)
I0619 00:11:56.358841 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0c1f35568fe24c6c8cd2b88871b4520d on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 8140397d408e47129a182fb6255a0a39)
I0619 00:11:56.359144 25085 catalog_manager.cc:4596] ReplaceTablet: tablet 0c1f35568fe24c6c8cd2b88871b4520d deleted and replaced by tablet 8140397d408e47129a182fb6255a0a39
I0619 00:11:56.360851 25080 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_951214baec60486e8b27aec35a5923e6" } new_table_name: "default.table_54f94ed54aa24b75bf10f8f57d8d1621" alter_external_catalogs: true
I0619 00:11:56.307544 25736 ts_tablet_manager.cc:1403] T a284677c4b7f46e18e25edee6024e3eb P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:56.363627 25730 tablet_replica.cc:261] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:56.363948 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:56.364202 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:56.364449 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:56.364708 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:56.365079 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:56.365320 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:56.366067 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:56.366286 25730 raft_consensus.cc:2042] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.366590 25730 raft_consensus.cc:2069] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.367854 25730 ts_tablet_manager.cc:1369] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.352505 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.368472 25599 tablet_service.cc:759] Processing CreateTablet for tablet 20a5866e22834304bb872bec30cb7ea4 (table=default.table_786d8b2d45904f2da470d39c2cb50010 [id=87ddbaef84354ac791a9feaf7bf65934]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.357946 25326 tablet_service.cc:800] Processing DeleteTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.361315 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 0c1f35568fe24c6c8cd2b88871b4520d with delete_type TABLET_DATA_DELETED (replaced by tablet 8140397d408e47129a182fb6255a0a39) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.361312 25323 tablet_service.cc:759] Processing CreateTablet for tablet 20a5866e22834304bb872bec30cb7ea4 (table=default.table_786d8b2d45904f2da470d39c2cb50010 [id=87ddbaef84354ac791a9feaf7bf65934]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.362828 25330 tablet_service.cc:759] Processing CreateTablet for tablet 89e2ca23c88748c68f0b30e10a899244 (table=default.table_c1b38858230546e1a993f452735a5f78 [id=5d6a973ecdd642eb96b94a470916a802]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.369091 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 20a5866e22834304bb872bec30cb7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.369251 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 20a5866e22834304bb872bec30cb7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.369148 25600 tablet_service.cc:759] Processing CreateTablet for tablet 89e2ca23c88748c68f0b30e10a899244 (table=default.table_c1b38858230546e1a993f452735a5f78 [id=5d6a973ecdd642eb96b94a470916a802]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.369561 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 89e2ca23c88748c68f0b30e10a899244. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.369746 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 89e2ca23c88748c68f0b30e10a899244. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.372061 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 0c1f35568fe24c6c8cd2b88871b4520d with delete_type TABLET_DATA_DELETED (replaced by tablet 8140397d408e47129a182fb6255a0a39) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.374375 25069 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC with a delay of 277 ms (attempt = 5)
W0619 00:11:56.374671 25069 catalog_manager.cc:2910] Async tablet task 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d44681acacf4f04b6fcfbd4c9a18e87 consensus state has no leader
I0619 00:11:56.375972 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 270) took 2632ms. Request Metrics: {"tablet-delete.queue_time_us":2601833}
I0619 00:11:56.376636 25724 ts_tablet_manager.cc:1073] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.033s	user 0.004s	sys 0.000s
I0619 00:11:56.376765 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 272) took 2558ms. Request Metrics: {"tablet-delete.queue_time_us":2557576}
I0619 00:11:56.377037 25724 ts_tablet_manager.cc:1028] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:56.377413 25724 tablet_bootstrap.cc:427] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:56.377429 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 273) took 2518ms. Request Metrics: {"tablet-delete.queue_time_us":2517302}
I0619 00:11:56.377818 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d2a13e99b1964be7a97603e011c9cb6b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:56.378867 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d2a13e99b1964be7a97603e011c9cb6b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.379238 25067 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 303 ms (attempt = 5)
I0619 00:11:56.379547 25724 tablet_bootstrap.cc:578] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:56.379568 25067 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
W0619 00:11:56.379967 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet edfdedbd4eca4b62be24b1649f820d0e with error code TABLET_NOT_RUNNING: Already present: State transition of tablet edfdedbd4eca4b62be24b1649f820d0e already in progress: creating tablet
I0619 00:11:56.380209 25067 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 39 ms (attempt = 2)
W0619 00:11:56.380658 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:56.381358 25600 ts_tablet_manager.cc:1173] T 89e2ca23c88748c68f0b30e10a899244 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.379997 26008 raft_consensus.cc:457] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.381742 26008 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.138s (starting election)
I0619 00:11:56.381995 26008 raft_consensus.cc:479] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.382917 26008 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.383307 26008 leader_election.cc:243] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:56.383239 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet e9a221a1d9814ef99a0b4ccaa04d1017 already in progress: creating tablet
I0619 00:11:56.384451 25067 catalog_manager.cc:2891] Scheduling retry of e9a221a1d9814ef99a0b4ccaa04d1017 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 24 ms (attempt = 1)
I0619 00:11:56.378916 25330 ts_tablet_manager.cc:1173] T 89e2ca23c88748c68f0b30e10a899244 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.384783 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:56.385033 25323 ts_tablet_manager.cc:1173] T 20a5866e22834304bb872bec30cb7ea4 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.385320 25623 raft_consensus.cc:2175] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:56.386865 25290 leader_election.cc:393] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:56.379488 25727 raft_consensus.cc:325] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.389504 25727 raft_consensus.cc:351] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.389752 25727 raft_consensus.cc:628] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:56.392655 25601 ts_tablet_manager.cc:1173] T 2a49647c4d13452eac7a2747e4c62040 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.390208 25727 consensus_queue.cc:227] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.398833 25727 ts_tablet_manager.cc:1073] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.078s	user 0.004s	sys 0.000s
W0619 00:11:56.419493 25069 catalog_manager.cc:2910] Async tablet task eb36a22cf80841b58fd5ce9e6998bf00 (table default.table_c553f0db13024123bd9fe8fb7de37a72 [id=fca6fa877b5f4226858ebe70c5d90418]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet eb36a22cf80841b58fd5ce9e6998bf00 consensus state has no leader
I0619 00:11:56.419539 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:11:56.420572 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:56.425341 25727 ts_tablet_manager.cc:1028] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:56.425437 25330 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.425753 25727 tablet_bootstrap.cc:427] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:56.427451 25079 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:56.426949 26092 raft_consensus.cc:457] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.428596 26092 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.086s (starting election)
I0619 00:11:56.428248 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.428983 26092 raft_consensus.cc:479] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.433706 25323 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:56.425200 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 197687us
I0619 00:11:56.434861 26092 leader_election.cc:243] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.435267 26092 leader_election.cc:243] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.436995 25599 ts_tablet_manager.cc:1173] T 20a5866e22834304bb872bec30cb7ea4 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.437284 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 251) took 2935ms. Request Metrics: {"tablet-delete.queue_time_us":2593529}
I0619 00:11:56.438102 25467 ts_tablet_manager.cc:1173] T 26455f6f1a47489cab7576d428344fd8 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.438755 25736 tablet_replica.cc:261] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:56.438999 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:56.439211 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:56.439399 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:56.439576 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:56.439810 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:56.439997 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:56.440172 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:56.440345 25736 raft_consensus.cc:2042] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.440608 25736 raft_consensus.cc:2069] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.441284 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.442850 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.444952 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.445811 25071 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 1024 ms (attempt = 7)
W0619 00:11:56.446146 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 consensus state has no leader
I0619 00:11:56.447496 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.448284 25736 ts_tablet_manager.cc:1369] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.448640 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "86959e7dda374ba9ac23d3c1fd30d993" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.449606 25489 raft_consensus.cc:2206] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.449504 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_07a87b17722646a38c8a8cd224140386"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:56.450263 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:56.450449 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_07a87b17722646a38c8a8cd224140386 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.451723 25724 tablet_bootstrap.cc:427] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:56.452014 25725 tablet_bootstrap.cc:427] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:56.452113 25724 ts_tablet_manager.cc:1045] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.075s	user 0.005s	sys 0.004s
I0619 00:11:56.452347 25725 ts_tablet_manager.cc:1045] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.190s	user 0.004s	sys 0.003s
I0619 00:11:56.453575 25724 raft_consensus.cc:325] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.454082 25725 raft_consensus.cc:325] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.455175 25725 raft_consensus.cc:351] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.455363 25725 raft_consensus.cc:628] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:56.455806 25725 consensus_queue.cc:227] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.460561 25725 ts_tablet_manager.cc:1073] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.001s
I0619 00:11:56.460968 25725 ts_tablet_manager.cc:1028] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:56.461383 25725 tablet_bootstrap.cc:427] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:56.465759 25725 tablet_bootstrap.cc:578] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:56.467186 25079 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:56.467944 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.468667 25290 leader_election.cc:393] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.468950 25290 leader_election.cc:270] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.468921 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3e7724f967ae41d7b523055d1a4f58db" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.469516 25489 raft_consensus.cc:2206] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.470988 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.472192 26008 raft_consensus.cc:2732] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.087s (election complete)
I0619 00:11:56.472434 26008 raft_consensus.cc:2784] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.467116 26145 raft_consensus.cc:457] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.477923 26008 raft_consensus.cc:2487] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:56.482260 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.483427 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 540 ms (attempt = 6)
W0619 00:11:56.483739 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:56.485837 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.486021 25290 leader_election.cc:393] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.486284 25290 leader_election.cc:270] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.458719 25730 ts_tablet_manager.cc:1382] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:56.469967 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.490437 26124 raft_consensus.cc:2732] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.139s (election complete)
I0619 00:11:56.490667 26124 raft_consensus.cc:2784] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.491410 25323 tablet_service.cc:759] Processing CreateTablet for tablet d7c7c5fb6fba42928768ef0e83abdbe1 (table=default.table_c053afda87e949b588e952600c8a012c [id=d934d712c973454099670fb2302a8c76]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.492036 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d7c7c5fb6fba42928768ef0e83abdbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.494778 25730 log.cc:986] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/54adb588955b45b79de9752f254728ea
I0619 00:11:56.494982 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19ad911ec6704aef85d9b7f172f08bdc" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.495342 25330 tablet_service.cc:759] Processing CreateTablet for tablet c1223a8bb4fc4714b962ea5a49903675 (table=default.table_4f9bc225373c46aaa2367ff751ca3698 [id=d0199d2e634b4083903702b3b85a3141]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.495455 25730 ts_tablet_manager.cc:1403] T 54adb588955b45b79de9752f254728ea P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:56.497831 26124 raft_consensus.cc:2487] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.501346 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c1223a8bb4fc4714b962ea5a49903675. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.499300 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 200) took 2467ms. Request Metrics: {"tablet-delete.queue_time_us":2331231}
I0619 00:11:56.503370 25727 tablet_bootstrap.cc:578] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:56.504124 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 204) took 2297ms. Request Metrics: {"tablet-delete.queue_time_us":2296424}
I0619 00:11:56.446125 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 274) took 2582ms. Request Metrics: {"tablet-delete.queue_time_us":2517601}
I0619 00:11:56.504946 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 541 ms (attempt = 6)
I0619 00:11:56.505180 25741 tablet_replica.cc:261] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
W0619 00:11:56.505242 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 consensus state has no leader
I0619 00:11:56.505415 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:56.505621 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:56.505852 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:56.506050 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:56.506340 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(4a88e394b199431e8b794a1b1aec847f)
I0619 00:11:56.506556 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(4a88e394b199431e8b794a1b1aec847f)
W0619 00:11:56.506641 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 93930ffe285d48d080a8ca4793e871d9 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 93930ffe285d48d080a8ca4793e871d9 already in progress: creating tablet
I0619 00:11:56.506778 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(4a88e394b199431e8b794a1b1aec847f)
W0619 00:11:56.506837 25290 consensus_peers.cc:434] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet a4c653a255984d3bac88df63362b9334. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:56.506989 25067 catalog_manager.cc:2891] Scheduling retry of 93930ffe285d48d080a8ca4793e871d9 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 23 ms (attempt = 1)
I0619 00:11:56.509850 25323 ts_tablet_manager.cc:1173] T d7c7c5fb6fba42928768ef0e83abdbe1 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.510761 25079 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_a822c7bace2a4cfd9c1c7f0a9c992551" } new_table_name: "default.table_fa2ab9b9fd3048349a823c382aad8b5c" alter_external_catalogs: true
W0619 00:11:56.514652 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.520174 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 264 ms (attempt = 5)
W0619 00:11:56.520478 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:56.520748 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.522382 25741 raft_consensus.cc:2042] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.522701 25741 raft_consensus.cc:2069] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.523910 25741 ts_tablet_manager.cc:1369] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.526170 25730 tablet_replica.cc:261] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:56.526445 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
W0619 00:11:56.508733 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.525676 25724 raft_consensus.cc:351] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.527011 25079 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_fa2ab9b9fd3048349a823c382aad8b5c [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 19 ms (attempt = 1)
I0619 00:11:56.527163 25724 raft_consensus.cc:628] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:56.527386 25079 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:56.527904 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet a284677c4b7f46e18e25edee6024e3eb (table default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]) successfully deleted
I0619 00:11:56.528396 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) successfully deleted
I0619 00:11:56.528300 25724 consensus_queue.cc:227] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.529359 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
W0619 00:11:56.529714 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet e60ecc51c92c461889b360dde1dc5cf2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet e60ecc51c92c461889b360dde1dc5cf2 already in progress: creating tablet
I0619 00:11:56.530069 25599 tablet_service.cc:759] Processing CreateTablet for tablet d7c7c5fb6fba42928768ef0e83abdbe1 (table=default.table_c053afda87e949b588e952600c8a012c [id=d934d712c973454099670fb2302a8c76]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.530766 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d7c7c5fb6fba42928768ef0e83abdbe1. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:56.507263 25423 leader_election.cc:306] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:11:56.537853 25423 leader_election.cc:270] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.537963 25489 raft_consensus.cc:1081] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:56.539011 25727 tablet_bootstrap.cc:427] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:56.539407 25727 ts_tablet_manager.cc:1045] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.114s	user 0.005s	sys 0.000s
I0619 00:11:56.539494 25724 ts_tablet_manager.cc:1073] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.087s	user 0.003s	sys 0.001s
I0619 00:11:56.539860 25724 ts_tablet_manager.cc:1028] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:56.540194 25724 tablet_bootstrap.cc:427] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:56.504521 25419 leader_election.cc:306] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.474088 26145 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.027s (starting election)
I0619 00:11:56.540952 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:56.541623 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:56.541079 25727 raft_consensus.cc:325] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.541836 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:56.541805 26008 consensus_queue.cc:963] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:56.542068 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:56.542038 25727 raft_consensus.cc:351] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.542304 25727 raft_consensus.cc:628] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:56.542268 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:56.542886 26100 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.680s (election complete)
I0619 00:11:56.542768 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:56.543126 26100 raft_consensus.cc:2487] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.543161 25730 raft_consensus.cc:2042] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.541149 26145 raft_consensus.cc:479] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.544634 26145 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.545087 26145 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.546152 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.548272 26140 sys_catalog.cc:188] Verifying existing consensus state
I0619 00:11:56.550107 25467 tablet_service.cc:759] Processing CreateTablet for tablet 2a49647c4d13452eac7a2747e4c62040 (table=default.table_b75c9264a29e43658fcbd7c31b36636a [id=b1ae4aee174a4d2b9cdf02b69bf3a8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.550673 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2a49647c4d13452eac7a2747e4c62040. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:56.550889 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:11:56.551024 25419 leader_election.cc:306] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.551410 25730 raft_consensus.cc:2069] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.553786 25730 ts_tablet_manager.cc:1369] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.558017 26089 raft_consensus.cc:457] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.558261 26089 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.839s (starting election)
I0619 00:11:56.558357 25725 tablet_bootstrap.cc:427] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:56.558712 25725 ts_tablet_manager.cc:1045] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.097s	user 0.005s	sys 0.006s
I0619 00:11:56.558501 26089 raft_consensus.cc:479] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.559470 26089 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.559870 26089 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.560035 25725 raft_consensus.cc:325] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.560420 26008 raft_consensus.cc:457] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.560751 26008 raft_consensus.cc:2732] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.695s (starting election)
I0619 00:11:56.560993 26008 raft_consensus.cc:479] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.561825 25724 tablet_bootstrap.cc:578] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:56.562021 26008 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.561928 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.562443 26008 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:56.563997 25553 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:56.565738 25290 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: d62715e230154e2abe126ef97feec7ce
W0619 00:11:56.566368 25290 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: d62715e230154e2abe126ef97feec7ce
I0619 00:11:56.567101 25290 leader_election.cc:270] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.569278 26008 raft_consensus.cc:2732] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.708s (election complete)
I0619 00:11:56.569531 26008 raft_consensus.cc:2487] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.573900 25620 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19ad911ec6704aef85d9b7f172f08bdc" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:56.574385 25620 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.871s (vote granted)
I0619 00:11:56.574607 25620 raft_consensus.cc:2245] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
W0619 00:11:56.576205 25558 leader_election.cc:306] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.579157 25071 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 38 ms (attempt = 1)
I0619 00:11:56.531972 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.534128 25080 catalog_manager.cc:3663] Got report from deleted tablet b34043f14f1c4edd97a1d4689ced5b32 (table default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:56.580317 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:56.551966 25069 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC with a delay of 1031 ms (attempt = 7)
W0619 00:11:56.581055 25069 catalog_manager.cc:2910] Async tablet task 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:56.530237 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_fb371427de4e44609a320dc1176820cb"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:56.581857 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fb371427de4e44609a320dc1176820cb in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.583335 25080 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:56.583997 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
W0619 00:11:56.583425 25071 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 53745.4us
I0619 00:11:56.586447 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 537 ms (attempt = 6)
W0619 00:11:56.586730 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:56.587009 26140 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Bootstrap starting.
I0619 00:11:56.589975 25080 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:56.590276 25080 catalog_manager.cc:3663] Got report from deleted tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1): Sending delete request for this tablet
I0619 00:11:56.592710 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.593061 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3537c79c30a1401e8a78b387f08eff58 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet d0624684b6b04ba3be260b2705dc00e1)
I0619 00:11:56.602927 25080 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:56.604220 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:56.605252 25326 tablet_service.cc:759] Processing CreateTablet for tablet 8140397d408e47129a182fb6255a0a39 (table=default.table_a9a83994991e4fcf9b36cb9a3d0590b0 [id=1bb609c5a8114109b7cfcfd006a2143f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.605895 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8140397d408e47129a182fb6255a0a39. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.583878 25727 consensus_queue.cc:227] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.608512 25727 ts_tablet_manager.cc:1073] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.069s	user 0.004s	sys 0.000s
I0619 00:11:56.608850 25727 ts_tablet_manager.cc:1028] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:56.609613 25727 tablet_bootstrap.cc:427] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:56.611932 25079 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:56.613420 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:56.614204 25724 tablet_bootstrap.cc:427] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:56.614745 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_fa2ab9b9fd3048349a823c382aad8b5c [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 42 ms (attempt = 2)
I0619 00:11:56.570147 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
W0619 00:11:56.615018 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_fa2ab9b9fd3048349a823c382aad8b5c [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:56.570551 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e5e691e6d374cd4bba1f18d8c821723" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.614784 25724 ts_tablet_manager.cc:1045] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.075s	user 0.004s	sys 0.001s
I0619 00:11:56.615985 25323 tablet_service.cc:759] Processing CreateTablet for tablet bda4e1ed3e8441ba90f63b78f67c61c5 (table=default.table_ff6a2c70c1d94d7a814d423d400f6c33 [id=5df0dc357def442bab8703e58586ecef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.616585 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bda4e1ed3e8441ba90f63b78f67c61c5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.616977 25601 tablet_service.cc:759] Processing CreateTablet for tablet c1223a8bb4fc4714b962ea5a49903675 (table=default.table_4f9bc225373c46aaa2367ff751ca3698 [id=d0199d2e634b4083903702b3b85a3141]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.617010 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_6ace57434b5243c7bd22bd60752a065d" }
I0619 00:11:56.617558 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c1223a8bb4fc4714b962ea5a49903675. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.617633 25602 tablet_service.cc:759] Processing CreateTablet for tablet 8140397d408e47129a182fb6255a0a39 (table=default.table_a9a83994991e4fcf9b36cb9a3d0590b0 [id=1bb609c5a8114109b7cfcfd006a2143f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.618314 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8140397d408e47129a182fb6255a0a39. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.620012 25600 tablet_service.cc:759] Processing CreateTablet for tablet bda4e1ed3e8441ba90f63b78f67c61c5 (table=default.table_ff6a2c70c1d94d7a814d423d400f6c33 [id=5df0dc357def442bab8703e58586ecef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.620573 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bda4e1ed3e8441ba90f63b78f67c61c5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.623741 25598 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.624334 25326 ts_tablet_manager.cc:1173] T 8140397d408e47129a182fb6255a0a39 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.626031 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:56.616202 25725 raft_consensus.cc:351] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.626745 25725 raft_consensus.cc:628] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:56.612984 25468 tablet_service.cc:759] Processing CreateTablet for tablet 0c1f35568fe24c6c8cd2b88871b4520d (table=default.table_a9a83994991e4fcf9b36cb9a3d0590b0 [id=1bb609c5a8114109b7cfcfd006a2143f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.626838 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_9fa9d2f02d024d7592c83d283fbdff04" }
I0619 00:11:56.627336 25725 consensus_queue.cc:227] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.629017 26100 raft_consensus.cc:457] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.613545 25462 tablet_service.cc:800] Processing DeleteTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.629238 25725 ts_tablet_manager.cc:1073] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.070s	user 0.004s	sys 0.000s
I0619 00:11:56.586220 25470 tablet_service.cc:759] Processing CreateTablet for tablet beddf1523a854f5b9292c0e4013d7436 (table=default.table_a822c7bace2a4cfd9c1c7f0a9c992551 [id=0d90b232db8e48ed9bfab0c59c6fbdba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.630194 25330 ts_tablet_manager.cc:1173] T c1223a8bb4fc4714b962ea5a49903675 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.631801 25331 tablet_service.cc:759] Processing CreateTablet for tablet a5c3203e9b6d423f810e1106369da3d7 (table=default.table_9fa9d2f02d024d7592c83d283fbdff04 [id=a78fb9bff11148f2857cfb4e2fda9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.629240 26100 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.988s (starting election)
I0619 00:11:56.632392 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a5c3203e9b6d423f810e1106369da3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.632304 26100 raft_consensus.cc:479] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.633263 26100 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.633656 26100 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.614259 25558 leader_election.cc:270] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.635555 26151 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.815s (election complete)
I0619 00:11:56.635583 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.635884 26151 raft_consensus.cc:2487] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:56.636399 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 60212.1us
I0619 00:11:56.636775 25600 ts_tablet_manager.cc:1173] T bda4e1ed3e8441ba90f63b78f67c61c5 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.637048 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
W0619 00:11:56.637028 25419 leader_election.cc:306] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.640043 25087 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:56.640702 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:56.641441 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:56.641947 25624 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.904s (vote granted)
I0619 00:11:56.642180 25331 ts_tablet_manager.cc:1173] T a5c3203e9b6d423f810e1106369da3d7 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.642199 25624 raft_consensus.cc:2245] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
W0619 00:11:56.646304 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.646760 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.646994 25601 ts_tablet_manager.cc:1173] T c1223a8bb4fc4714b962ea5a49903675 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.647584 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 9d44681acacf4f04b6fcfbd4c9a18e87
I0619 00:11:56.647912 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.648288 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.648548 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.650893 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.651520 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:56.652670 25069 catalog_manager.cc:2910] Async tablet task 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d44681acacf4f04b6fcfbd4c9a18e87 consensus state has no leader
I0619 00:11:56.638675 25724 raft_consensus.cc:325] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.653300 25724 raft_consensus.cc:351] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.653517 25724 raft_consensus.cc:628] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:56.654065 25724 consensus_queue.cc:227] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.655664 25724 ts_tablet_manager.cc:1073] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.040s	user 0.003s	sys 0.001s
I0619 00:11:56.655877 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.655970 25724 ts_tablet_manager.cc:1028] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:56.656345 25724 tablet_bootstrap.cc:427] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:56.656664 26151 raft_consensus.cc:457] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.656945 26151 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.162s (starting election)
I0619 00:11:56.657232 26151 raft_consensus.cc:479] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.658241 26151 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.658640 26151 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.659330 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet a5c3203e9b6d423f810e1106369da3d7
I0619 00:11:56.659621 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5c3203e9b6d423f810e1106369da3d7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.660018 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5c3203e9b6d423f810e1106369da3d7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.660207 25068 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 1026 ms (attempt = 7)
I0619 00:11:56.660378 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5c3203e9b6d423f810e1106369da3d7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:11:56.660477 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:56.662205 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.662693 25602 ts_tablet_manager.cc:1173] T 8140397d408e47129a182fb6255a0a39 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:56.663537 25553 leader_election.cc:306] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.663745 25599 ts_tablet_manager.cc:1173] T d7c7c5fb6fba42928768ef0e83abdbe1 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.664610 25736 ts_tablet_manager.cc:1382] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:56.664726 25087 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:56.664902 25727 tablet_bootstrap.cc:578] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:56.664966 25736 log.cc:986] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/54373917567a479d93dc8a3fd006d2fe
I0619 00:11:56.665370 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:56.665482 25736 ts_tablet_manager.cc:1403] T 54373917567a479d93dc8a3fd006d2fe P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:56.578863 25467 ts_tablet_manager.cc:1173] T 2a49647c4d13452eac7a2747e4c62040 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.666560 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.666970 25084 master_service.cc:523] ReplaceTablet: received request to replace tablet 0fee3be144c14edab6db58c99e3cda7d from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:56.666759 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_25bd5ae167b74dea974c7eabdd1cd7b0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:56.667842 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_25bd5ae167b74dea974c7eabdd1cd7b0 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.613981 25423 leader_election.cc:379] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.668128 25423 leader_election.cc:270] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:56.668618 26146 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.925s (election complete)
I0619 00:11:56.668893 26146 raft_consensus.cc:2539] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:56.669203 26146 raft_consensus.cc:593] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:56.669684 26146 consensus_queue.cc:204] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.672107 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_fa2ab9b9fd3048349a823c382aad8b5c [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 68 ms (attempt = 3)
W0619 00:11:56.672399 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_fa2ab9b9fd3048349a823c382aad8b5c [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:56.675367 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 2056 ms (attempt = 8)
W0619 00:11:56.675622 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:56.676007 25730 ts_tablet_manager.cc:1382] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I0619 00:11:56.676270 25730 log.cc:986] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/ee4c4769f75141f9b2ef864ab8a8d9e0
I0619 00:11:56.676674 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.676875 25730 ts_tablet_manager.cc:1403] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:56.678266 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.680995 25600 tablet_service.cc:800] Processing DeleteTablet for tablet a5c3203e9b6d423f810e1106369da3d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.681433 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.686087 25336 tablet_service.cc:800] Processing DeleteTablet for tablet a5c3203e9b6d423f810e1106369da3d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.685860 25741 ts_tablet_manager.cc:1382] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:56.688102 25067 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 552 ms (attempt = 6)
W0619 00:11:56.688547 25067 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
W0619 00:11:56.689141 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 75178.8us
I0619 00:11:56.689281 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 252) took 3178ms. Request Metrics: {"tablet-delete.queue_time_us":2926985}
I0619 00:11:56.690073 25602 tablet_service.cc:759] Processing CreateTablet for tablet a5c3203e9b6d423f810e1106369da3d7 (table=default.table_9fa9d2f02d024d7592c83d283fbdff04 [id=a78fb9bff11148f2857cfb4e2fda9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.690052 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 263) took 2863ms. Request Metrics: {"tablet-delete.queue_time_us":2862781}
I0619 00:11:56.690704 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a5c3203e9b6d423f810e1106369da3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.690656 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 268) took 2744ms. Request Metrics: {"tablet-delete.queue_time_us":2718042}
I0619 00:11:56.691241 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 269) took 2688ms. Request Metrics: {"tablet-delete.queue_time_us":2681146}
I0619 00:11:56.691854 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 270) took 2665ms. Request Metrics: {"tablet-delete.queue_time_us":2664788}
I0619 00:11:56.692370 25724 tablet_bootstrap.cc:578] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:56.692827 25599 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.695757 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.696589 25725 ts_tablet_manager.cc:1028] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:56.697005 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.697695 25725 tablet_bootstrap.cc:427] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:56.702924 25558 leader_election.cc:306] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.703552 25558 leader_election.cc:270] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.704417 26151 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.659s (election complete)
I0619 00:11:56.704640 26151 raft_consensus.cc:2487] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:56.706089 25558 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.706589 25558 leader_election.cc:270] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.707345 26151 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.148s (election complete)
I0619 00:11:56.707562 26151 raft_consensus.cc:2487] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.698271 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0c1f35568fe24c6c8cd2b88871b4520d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.710304 25323 ts_tablet_manager.cc:1173] T bda4e1ed3e8441ba90f63b78f67c61c5 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.710642 25069 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 2074 ms (attempt = 8)
W0619 00:11:56.710985 25069 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:56.711448 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet beddf1523a854f5b9292c0e4013d7436. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:56.713131 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.700256 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.704128 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 0c1f35568fe24c6c8cd2b88871b4520d with delete_type TABLET_DATA_DELETED (replaced by tablet 8140397d408e47129a182fb6255a0a39) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.692450 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 271) took 2634ms. Request Metrics: {"tablet-delete.queue_time_us":2632965}
I0619 00:11:56.701009 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.716261 25736 tablet_replica.cc:261] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:56.716614 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet 3722b203a516466489c74ef6013eacae from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:56.717665 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.720111 25468 ts_tablet_manager.cc:1173] T 0c1f35568fe24c6c8cd2b88871b4520d P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.720842 26146 raft_consensus.cc:457] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.721120 26146 raft_consensus.cc:2732] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.867s (starting election)
I0619 00:11:56.721387 26146 raft_consensus.cc:479] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.716775 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:56.722411 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:56.722982 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:56.724328 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:56.724263 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 2050 ms (attempt = 8)
I0619 00:11:56.724380 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 54373917567a479d93dc8a3fd006d2fe (table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]) successfully deleted
W0619 00:11:56.724536 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:56.724936 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:56.725430 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:56.725716 25081 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_fa2ab9b9fd3048349a823c382aad8b5c" } new_table_name: "default.table_2d9df18dc333494d9398874c17e04acf" alter_external_catalogs: true
I0619 00:11:56.722396 26146 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.726851 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(edfdedbd4eca4b62be24b1649f820d0e)
I0619 00:11:56.729259 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "edfdedbd4eca4b62be24b1649f820d0e" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.729861 25736 raft_consensus.cc:2042] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.726955 26146 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.725973 25725 tablet_bootstrap.cc:578] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:56.730351 25727 tablet_bootstrap.cc:427] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:56.730594 25351 raft_consensus.cc:2784] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.730913 25727 ts_tablet_manager.cc:1045] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.121s	user 0.005s	sys 0.000s
W0619 00:11:56.731904 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.732343 25736 raft_consensus.cc:2069] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.733211 25727 raft_consensus.cc:325] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:56.734041 25419 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.687767 25741 log.cc:986] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/4a88e394b199431e8b794a1b1aec847f
I0619 00:11:56.735162 25741 ts_tablet_manager.cc:1403] T 4a88e394b199431e8b794a1b1aec847f P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:56.708850 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 203) took 2502ms. Request Metrics: {"mutex_wait_us":2121,"tablet-delete.queue_time_us":2315135}
I0619 00:11:56.737610 25351 raft_consensus.cc:1081] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:56.737363 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 275) took 2797ms. Request Metrics: {"mutex_wait_us":8,"tablet-delete.queue_time_us":2565134}
I0619 00:11:56.738531 25741 tablet_replica.cc:261] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:56.738550 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 4a88e394b199431e8b794a1b1aec847f (table default.table_2c7f2c73082443ab869e2910097fa796 [id=9b96421847354087b7a4fa2afc633be3]) successfully deleted
I0619 00:11:56.738685 25730 tablet_replica.cc:261] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:56.739171 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:56.739428 25087 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:56.739640 25624 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.678s (vote granted)
I0619 00:11:56.739882 25624 raft_consensus.cc:2245] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:56.740118 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:56.740816 25085 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:56.741135 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 0fee3be144c14edab6db58c99e3cda7d
I0619 00:11:56.741137 25085 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:56.741468 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0fee3be144c14edab6db58c99e3cda7d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 6755b36ed905430e8724d43920c9c0d8)
I0619 00:11:56.741647 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 2084 ms (attempt = 8)
W0619 00:11:56.741940 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:56.742259 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:56.742511 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:56.742727 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:56.742921 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:56.743011 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:56.743152 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:56.743367 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:56.743387 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:56.743788 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0fee3be144c14edab6db58c99e3cda7d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 6755b36ed905430e8724d43920c9c0d8)
I0619 00:11:56.744168 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0fee3be144c14edab6db58c99e3cda7d on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 6755b36ed905430e8724d43920c9c0d8)
I0619 00:11:56.744513 25084 catalog_manager.cc:4596] ReplaceTablet: tablet 0fee3be144c14edab6db58c99e3cda7d deleted and replaced by tablet 6755b36ed905430e8724d43920c9c0d8
I0619 00:11:56.745998 25085 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:56.746676 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
W0619 00:11:56.725667 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f75b8a4f1fd845fbab6f83f59b7cfc58 already in progress: creating tablet
I0619 00:11:56.747182 25071 catalog_manager.cc:2891] Scheduling retry of f75b8a4f1fd845fbab6f83f59b7cfc58 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 51 ms (attempt = 1)
I0619 00:11:56.748311 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_fa2ab9b9fd3048349a823c382aad8b5c [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 134 ms (attempt = 4)
W0619 00:11:56.748960 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_fa2ab9b9fd3048349a823c382aad8b5c [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:56.749076 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 0fee3be144c14edab6db58c99e3cda7d with delete_type TABLET_DATA_DELETED (replaced by tablet 6755b36ed905430e8724d43920c9c0d8) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.749716 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:56.749755 25336 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.749253 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) successfully deleted
I0619 00:11:56.749928 25730 raft_consensus.cc:2042] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.750202 25730 raft_consensus.cc:2069] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.750234 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.750914 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3031f8f5176e4479af3b3f3d51ebdb06"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:56.751634 25081 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 62 ms (attempt = 1)
W0619 00:11:56.751899 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3031f8f5176e4479af3b3f3d51ebdb06 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:56.751929 25081 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
W0619 00:11:56.750316 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 because the tablet was not found. No further retry: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
W0619 00:11:56.753116 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7c57c5ca55f24621b66197a3698504bd because the tablet was not found. No further retry: Not found: Tablet not found: 7c57c5ca55f24621b66197a3698504bd
I0619 00:11:56.753451 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 0fee3be144c14edab6db58c99e3cda7d with delete_type TABLET_DATA_DELETED (replaced by tablet 6755b36ed905430e8724d43920c9c0d8) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.752851 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_947e41e56412414da7590c66fe5fc49a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:56.754231 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_947e41e56412414da7590c66fe5fc49a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:56.755221 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:56.734263 25727 raft_consensus.cc:351] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.757278 25727 raft_consensus.cc:628] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:56.757800 25727 consensus_queue.cc:227] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.761220 25470 ts_tablet_manager.cc:1173] T beddf1523a854f5b9292c0e4013d7436 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.763062 25736 ts_tablet_manager.cc:1369] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.764174 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.764261 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.765715 25730 ts_tablet_manager.cc:1369] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.769927 26008 raft_consensus.cc:457] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.770146 26008 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.044s (starting election)
I0619 00:11:56.770385 26008 raft_consensus.cc:479] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.771379 26008 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.771741 25354 raft_consensus.cc:1081] T 3e7724f967ae41d7b523055d1a4f58db P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:56.771776 26008 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.773231 26146 consensus_queue.cc:963] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:56.775161 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.775732 25487 raft_consensus.cc:2206] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.775924 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.776160 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.776557 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.776818 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.777017 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.777194 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.777359 25741 raft_consensus.cc:2042] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:56.778523 25741 raft_consensus.cc:2069] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.778167 25727 ts_tablet_manager.cc:1073] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.047s	user 0.004s	sys 0.000s
I0619 00:11:56.780742 25727 ts_tablet_manager.cc:1028] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:56.781080 25727 tablet_bootstrap.cc:427] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:56.781172 25470 tablet_service.cc:759] Processing CreateTablet for tablet 20a5866e22834304bb872bec30cb7ea4 (table=default.table_786d8b2d45904f2da470d39c2cb50010 [id=87ddbaef84354ac791a9feaf7bf65934]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.781950 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 20a5866e22834304bb872bec30cb7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.782548 26140 log.cc:525] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Log is configured to *not* fsync() on all Append() calls
I0619 00:11:56.783208 25727 tablet_bootstrap.cc:578] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:56.788060 25290 leader_election.cc:393] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.773758 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:56.777850 25622 raft_consensus.cc:1081] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:56.788550 25624 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.075s (vote granted)
I0619 00:11:56.782065 25724 tablet_bootstrap.cc:427] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
W0619 00:11:56.779770 25558 leader_election.cc:306] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.788811 25624 raft_consensus.cc:2245] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:56.788915 25558 leader_election.cc:270] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.789002 25724 ts_tablet_manager.cc:1045] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.133s	user 0.005s	sys 0.000s
I0619 00:11:56.789376 26151 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.863s (election complete)
I0619 00:11:56.789594 26151 raft_consensus.cc:2487] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:56.790729 25724 raft_consensus.cc:325] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.791303 25724 raft_consensus.cc:351] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.791471 25724 raft_consensus.cc:628] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:56.792184 25724 consensus_queue.cc:227] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.793804 25724 ts_tablet_manager.cc:1073] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I0619 00:11:56.764853 25084 catalog_manager.cc:3663] Got report from deleted tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
I0619 00:11:56.764807 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 3722b203a516466489c74ef6013eacae
I0619 00:11:56.785794 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 516 ms (attempt = 6)
W0619 00:11:56.794422 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:56.794162 25724 ts_tablet_manager.cc:1028] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:56.794492 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3722b203a516466489c74ef6013eacae on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet fe83202d55bc40e8ae193b9f6f7e9adf)
I0619 00:11:56.794430 25084 catalog_manager.cc:3663] Got report from deleted tablet edfdedbd4eca4b62be24b1649f820d0e (table default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:56.794996 25724 tablet_bootstrap.cc:427] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:56.795073 25084 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:56.795528 25290 leader_election.cc:379] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.795766 25290 leader_election.cc:270] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won.
I0619 00:11:56.796142 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:56.796203 26008 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.749s (election complete)
I0619 00:11:56.796425 26008 raft_consensus.cc:2539] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I0619 00:11:56.796507 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:56.796625 26008 raft_consensus.cc:457] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:56.796846 26008 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.066s (starting election)
I0619 00:11:56.796872 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:56.797042 26008 raft_consensus.cc:2784] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Advancing to term 2
I0619 00:11:56.797154 25724 tablet_bootstrap.cc:578] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:56.797888 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3722b203a516466489c74ef6013eacae on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet fe83202d55bc40e8ae193b9f6f7e9adf)
I0619 00:11:56.798286 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3722b203a516466489c74ef6013eacae on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet fe83202d55bc40e8ae193b9f6f7e9adf)
I0619 00:11:56.798635 25080 catalog_manager.cc:4596] ReplaceTablet: tablet 3722b203a516466489c74ef6013eacae deleted and replaced by tablet fe83202d55bc40e8ae193b9f6f7e9adf
I0619 00:11:56.800506 25725 tablet_bootstrap.cc:427] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:56.800899 25725 ts_tablet_manager.cc:1045] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.103s	user 0.004s	sys 0.002s
I0619 00:11:56.802296 25725 raft_consensus.cc:325] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.802939 25725 raft_consensus.cc:351] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.802742 26008 raft_consensus.cc:479] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.803133 25725 raft_consensus.cc:628] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:56.803719 26008 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.804070 26008 leader_election.cc:243] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.805269 25468 tablet_service.cc:759] Processing CreateTablet for tablet 89e2ca23c88748c68f0b30e10a899244 (table=default.table_c1b38858230546e1a993f452735a5f78 [id=5d6a973ecdd642eb96b94a470916a802]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.805822 26146 consensus_queue.cc:963] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:11:56.806478 25423 consensus_peers.cc:434] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet ee4c4769f75141f9b2ef864ab8a8d9e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:56.805925 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 89e2ca23c88748c68f0b30e10a899244. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.810809 25351 raft_consensus.cc:1049] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 2 FOLLOWER]: Rejecting Update request from peer 10bda0b4856f4441b5aed24cdc4764a8 for earlier term 1. Current term is 2. Ops: []
I0619 00:11:56.811687 25085 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:56.813017 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_54f94ed54aa24b75bf10f8f57d8d1621 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 2 got 0
I0619 00:11:56.813449 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 3722b203a516466489c74ef6013eacae with delete_type TABLET_DATA_DELETED (replaced by tablet fe83202d55bc40e8ae193b9f6f7e9adf) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.807798 26161 tablet.cc:1200] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:56.815258 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:56.815807 25487 raft_consensus.cc:2206] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.819783 25741 ts_tablet_manager.cc:1369] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:56.822420 25602 ts_tablet_manager.cc:1173] T a5c3203e9b6d423f810e1106369da3d7 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:56.822705 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.824652 25727 tablet_bootstrap.cc:427] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:56.824990 25727 ts_tablet_manager.cc:1045] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.044s	user 0.003s	sys 0.000s
I0619 00:11:56.826277 25727 raft_consensus.cc:325] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.826875 25727 raft_consensus.cc:351] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.827067 25727 raft_consensus.cc:628] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:56.829363 25290 leader_election.cc:393] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.830051 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.829919 26146 consensus_queue.cc:974] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:56.830567 26146 raft_consensus.cc:2779] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Stepping down as leader of term 1
I0619 00:11:56.803560 25725 consensus_queue.cc:227] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.830802 26146 raft_consensus.cc:628] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:56.831272 26146 consensus_queue.cc:227] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.831977 26146 consensus_peers.cc:456] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:56.832182 26146 consensus_peers.cc:456] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:56.832515 26146 raft_consensus.cc:2784] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Advancing to term 2
I0619 00:11:56.848539 25725 ts_tablet_manager.cc:1073] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.047s	user 0.005s	sys 0.000s
I0619 00:11:56.849244 25725 ts_tablet_manager.cc:1028] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:56.849781 25725 tablet_bootstrap.cc:427] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:56.851539 25323 tablet_service.cc:759] Processing CreateTablet for tablet 33783498947e4e038e876f7e409eea30 (table=default.table_07a87b17722646a38c8a8cd224140386 [id=e1814d17f14e45f4b45f1372005e6f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.852123 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 33783498947e4e038e876f7e409eea30. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.795087 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 1025 ms (attempt = 7)
W0619 00:11:56.854768 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:56.840271 25087 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]). Expected version 1 got 0
I0619 00:11:56.855484 25087 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 53 ms (attempt = 1)
W0619 00:11:56.855792 25087 catalog_manager.cc:3945] Failed to send 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:56.856133 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.856587 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 65 ms (attempt = 2)
W0619 00:11:56.856861 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
W0619 00:11:56.856998 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.857483 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_fb371427de4e44609a320dc1176820cb" } new_table_name: "default.table_9751fcfca6454dd99468df3ffb363e82" alter_external_catalogs: true
W0619 00:11:56.857830 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 4a88e394b199431e8b794a1b1aec847f because the tablet was not found. No further retry: Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:56.838757 25080 catalog_manager.cc:3663] Got report from deleted tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d): Sending delete request for this tablet
I0619 00:11:56.859467 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:56.859773 25084 catalog_manager.cc:3663] Got report from deleted tablet 3e7724f967ae41d7b523055d1a4f58db (table default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:56.860442 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.862552 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_35bd36a899a9473f80f5e06739f90c1b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:56.863442 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_35bd36a899a9473f80f5e06739f90c1b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:56.864012 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:56.868130 25087 catalog_manager.cc:2891] Scheduling retry of e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC with a delay of 26 ms (attempt = 1)
W0619 00:11:56.868410 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5051fc388d74ad39753c43b0ad25402 consensus state has no leader
I0619 00:11:56.872128 25087 catalog_manager.cc:3663] Got report from deleted tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d): Sending delete request for this tablet
I0619 00:11:56.872434 25087 catalog_manager.cc:3663] Got report from deleted tablet 3e7724f967ae41d7b523055d1a4f58db (table default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:56.873204 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 1066 ms (attempt = 7)
I0619 00:11:56.873299 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
W0619 00:11:56.874030 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:11:56.873654 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.879360 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_4f9bc225373c46aaa2367ff751ca3698" }
I0619 00:11:56.881155 25336 tablet_service.cc:759] Processing CreateTablet for tablet 992f255e14544af3b8a89cd295944f29 (table=default.table_d2a13e99b1964be7a97603e011c9cb6b [id=08e4e7f0b7864a91ad75b900b2a89186]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.881824 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 992f255e14544af3b8a89cd295944f29. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.881772 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_c8d425c090594b149578c26728881c9d" }
I0619 00:11:56.883280 25462 tablet_service.cc:759] Processing CreateTablet for tablet d7c7c5fb6fba42928768ef0e83abdbe1 (table=default.table_c053afda87e949b588e952600c8a012c [id=d934d712c973454099670fb2302a8c76]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.883895 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d7c7c5fb6fba42928768ef0e83abdbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.888499 25085 catalog_manager.cc:3663] Got report from deleted tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:56.889202 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:56.893283 25462 ts_tablet_manager.cc:1173] T d7c7c5fb6fba42928768ef0e83abdbe1 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.893488 25736 ts_tablet_manager.cc:1382] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:56.893771 25736 log.cc:986] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/edfdedbd4eca4b62be24b1649f820d0e
I0619 00:11:56.894237 25736 ts_tablet_manager.cc:1403] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:56.894798 26008 raft_consensus.cc:457] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.895043 26008 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.230s (starting election)
I0619 00:11:56.895037 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 272) took 2824ms. Request Metrics: {"tablet-delete.queue_time_us":2644491}
I0619 00:11:56.895465 25326 tablet_service.cc:759] Processing CreateTablet for tablet e5051fc388d74ad39753c43b0ad25402 (table=default.table_fb371427de4e44609a320dc1176820cb [id=8977b9567fe14118b3bc189f6581f264]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.895273 26008 raft_consensus.cc:479] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.896077 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5051fc388d74ad39753c43b0ad25402. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.896003 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 275 ms (attempt = 5)
W0619 00:11:56.896255 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:56.896268 26008 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.896576 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 1024 ms (attempt = 7)
I0619 00:11:56.896687 26008 leader_election.cc:243] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:56.896843 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:56.897569 25067 catalog_manager.cc:2891] Scheduling retry of e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC with a delay of 53 ms (attempt = 2)
W0619 00:11:56.898002 25067 catalog_manager.cc:2910] Async tablet task e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e5051fc388d74ad39753c43b0ad25402 consensus state has no leader
I0619 00:11:56.898176 25464 tablet_service.cc:759] Processing CreateTablet for tablet c1223a8bb4fc4714b962ea5a49903675 (table=default.table_4f9bc225373c46aaa2367ff751ca3698 [id=d0199d2e634b4083903702b3b85a3141]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.898746 25423 leader_election.cc:379] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.899005 25423 leader_election.cc:270] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:56.899538 26146 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.849s (election complete)
I0619 00:11:56.899845 26146 raft_consensus.cc:2539] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:56.900226 26146 raft_consensus.cc:457] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:56.900588 26146 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.645s (starting election)
I0619 00:11:56.900908 26146 raft_consensus.cc:2784] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.829151 26151 raft_consensus.cc:457] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.835170 25724 tablet_bootstrap.cc:427] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:56.904263 26151 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.514s (starting election)
I0619 00:11:56.903879 25727 consensus_queue.cc:227] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.904439 25724 ts_tablet_manager.cc:1045] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.110s	user 0.005s	sys 0.000s
I0619 00:11:56.858780 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 3722b203a516466489c74ef6013eacae with delete_type TABLET_DATA_DELETED (replaced by tablet fe83202d55bc40e8ae193b9f6f7e9adf) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:56.843184 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.860816 25601 tablet_service.cc:759] Processing CreateTablet for tablet 33783498947e4e038e876f7e409eea30 (table=default.table_07a87b17722646a38c8a8cd224140386 [id=e1814d17f14e45f4b45f1372005e6f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.879494 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.861796 25600 tablet_service.cc:759] Processing CreateTablet for tablet 992f255e14544af3b8a89cd295944f29 (table=default.table_d2a13e99b1964be7a97603e011c9cb6b [id=08e4e7f0b7864a91ad75b900b2a89186]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.904546 26151 raft_consensus.cc:479] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.905338 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet c1223a8bb4fc4714b962ea5a49903675
I0619 00:11:56.905655 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c1223a8bb4fc4714b962ea5a49903675 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.906018 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c1223a8bb4fc4714b962ea5a49903675 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.905617 26151 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.906368 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c1223a8bb4fc4714b962ea5a49903675 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.906427 26151 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.908700 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet e0070a115e8540cda5f999394deac616
I0619 00:11:56.908923 26151 raft_consensus.cc:457] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.909132 26151 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.637s (starting election)
I0619 00:11:56.909096 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 32 ms (attempt = 2)
W0619 00:11:56.909344 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:56.909360 26151 raft_consensus.cc:479] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.909737 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.910169 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:56.910267 26151 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.910143 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc7686daae274e0dba87703fe6924805" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.910645 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:11:56.911032 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:11:56.911432 25553 leader_election.cc:306] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.906657 25724 raft_consensus.cc:325] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.911700 25724 raft_consensus.cc:351] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:56.911885 25724 raft_consensus.cc:628] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:56.911921 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.912541 25624 raft_consensus.cc:1081] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:56.912770 25331 tablet_service.cc:800] Processing DeleteTablet for tablet c1223a8bb4fc4714b962ea5a49903675 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.912360 25724 consensus_queue.cc:227] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.913286 26151 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.913774 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "17e165e0995a4c029aa31392b9bbc04a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:56.914106 25724 ts_tablet_manager.cc:1073] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I0619 00:11:56.914201 25623 raft_consensus.cc:2784] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Advancing to term 2
I0619 00:11:56.914489 25623 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.257s (vote granted)
I0619 00:11:56.914973 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b34043f14f1c4edd97a1d4689ced5b32" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:56.916968 25553 leader_election.cc:306] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.917155 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 33783498947e4e038e876f7e409eea30. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.917676 25724 ts_tablet_manager.cc:1028] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:56.918654 25724 tablet_bootstrap.cc:427] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:56.921062 25084 catalog_manager.cc:3663] Got report from deleted tablet a284677c4b7f46e18e25edee6024e3eb (table default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:56.922562 25331 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.923316 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
W0619 00:11:56.923317 25290 leader_election.cc:306] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 54adb588955b45b79de9752f254728ea
I0619 00:11:56.899273 25725 tablet_bootstrap.cc:578] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:56.921838 25623 raft_consensus.cc:2245] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 2.
I0619 00:11:56.923905 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 992f255e14544af3b8a89cd295944f29. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.927356 25601 ts_tablet_manager.cc:1173] T 33783498947e4e038e876f7e409eea30 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.925354 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.932760 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 72 ms (attempt = 3)
W0619 00:11:56.933028 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:56.933143 25326 ts_tablet_manager.cc:1173] T e5051fc388d74ad39753c43b0ad25402 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.934761 25598 tablet_service.cc:800] Processing DeleteTablet for tablet c1223a8bb4fc4714b962ea5a49903675 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.935029 25727 ts_tablet_manager.cc:1073] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.110s	user 0.005s	sys 0.000s
I0619 00:11:56.935235 25599 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.935428 25598 tablet_service.cc:759] Processing CreateTablet for tablet e5051fc388d74ad39753c43b0ad25402 (table=default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.936015 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5051fc388d74ad39753c43b0ad25402. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.936028 25336 ts_tablet_manager.cc:1173] T 992f255e14544af3b8a89cd295944f29 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.937170 25323 ts_tablet_manager.cc:1173] T 33783498947e4e038e876f7e409eea30 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.937826 25727 ts_tablet_manager.cc:1028] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:56.938155 25727 tablet_bootstrap.cc:427] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:56.938866 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 2079 ms (attempt = 8)
W0619 00:11:56.939164 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:56.940748 26151 raft_consensus.cc:457] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:56.940865 25727 tablet_bootstrap.cc:578] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:56.940887 25084 catalog_manager.cc:3663] Got report from deleted tablet a284677c4b7f46e18e25edee6024e3eb (table default.table_7937f0a6914a40b8baff07c0983cb44c [id=07abf7ebedd843f094d8891ebbc73ecc]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:56.941009 26151 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.716s (starting election)
I0619 00:11:56.941550 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:56.941258 26151 raft_consensus.cc:479] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:56.942265 26151 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:56.942677 26151 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:56.943826 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f1b566b2b58f4955aa25ae990a4a8a1a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:56.944826 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 70 ms (attempt = 3)
I0619 00:11:56.945039 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
W0619 00:11:56.945077 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
W0619 00:11:56.945894 25553 leader_election.cc:306] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:56.946146 25467 tablet_service.cc:759] Processing CreateTablet for tablet 8140397d408e47129a182fb6255a0a39 (table=default.table_a9a83994991e4fcf9b36cb9a3d0590b0 [id=1bb609c5a8114109b7cfcfd006a2143f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.946604 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_54f94ed54aa24b75bf10f8f57d8d1621 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 2 got 1
I0619 00:11:56.946954 25087 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:56.946931 25462 tablet_service.cc:759] Processing CreateTablet for tablet bda4e1ed3e8441ba90f63b78f67c61c5 (table=default.table_ff6a2c70c1d94d7a814d423d400f6c33 [id=5df0dc357def442bab8703e58586ecef]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:56.947233 25087 catalog_manager.cc:3663] Got report from deleted tablet f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:56.850519 25730 ts_tablet_manager.cc:1382] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I0619 00:11:56.948935 25730 log.cc:986] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/075076aaa16e4ea8a596a7a8c52f103e
W0619 00:11:56.949224 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.949529 25730 ts_tablet_manager.cc:1403] T 075076aaa16e4ea8a596a7a8c52f103e P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:56.951007 25724 tablet_bootstrap.cc:578] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:56.951493 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 208) took 2694ms. Request Metrics: {"tablet-delete.queue_time_us":2480641}
I0619 00:11:56.953032 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.953192 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:56.960379 25598 ts_tablet_manager.cc:1173] T e5051fc388d74ad39753c43b0ad25402 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.962810 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:56.965471 25741 ts_tablet_manager.cc:1382] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:56.965765 25741 log.cc:986] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/3537c79c30a1401e8a78b387f08eff58
I0619 00:11:56.965978 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:56.966361 25741 ts_tablet_manager.cc:1403] T 3537c79c30a1401e8a78b387f08eff58 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:56.967419 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) successfully deleted
I0619 00:11:56.968323 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet edfdedbd4eca4b62be24b1649f820d0e (table default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]) successfully deleted
I0619 00:11:56.969437 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_10c2abdb4e2f45feb095ecf73709e89b" } new_table_name: "default.table_c74c58e16b0e4279ad767a97fe30c436" alter_external_catalogs: true
I0619 00:11:56.970080 25600 ts_tablet_manager.cc:1173] T 992f255e14544af3b8a89cd295944f29 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.970350 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:56.970765 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:56.897528 25470 ts_tablet_manager.cc:1173] T 20a5866e22834304bb872bec30cb7ea4 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.971192 25468 ts_tablet_manager.cc:1173] T 89e2ca23c88748c68f0b30e10a899244 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:56.971786 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c1223a8bb4fc4714b962ea5a49903675. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.972077 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 273) took 2899ms. Request Metrics: {"tablet-delete.queue_time_us":2821988}
I0619 00:11:56.972856 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 275) took 2861ms. Request Metrics: {"tablet-delete.queue_time_us":2860661}
I0619 00:11:56.973395 25736 tablet_replica.cc:261] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:56.973690 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.973891 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.974078 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.974279 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.974567 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.974884 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.974736 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "54adb588955b45b79de9752f254728ea" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.975097 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(3537c79c30a1401e8a78b387f08eff58)
I0619 00:11:56.975283 25487 raft_consensus.cc:2206] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.976766 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc7686daae274e0dba87703fe6924805" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:56.978262 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8140397d408e47129a182fb6255a0a39. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.980059 25067 catalog_manager.cc:2891] Scheduling retry of e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC with a delay of 101 ms (attempt = 3)
W0619 00:11:56.980324 25067 catalog_manager.cc:2910] Async tablet task e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e5051fc388d74ad39753c43b0ad25402 consensus state has no leader
W0619 00:11:56.981914 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.952183 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 209) took 2694ms. Request Metrics: {"tablet-delete.queue_time_us":2694290}
I0619 00:11:56.983057 25601 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:56.983441 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 211) took 2720ms. Request Metrics: {"tablet-delete.queue_time_us":2719980}
I0619 00:11:56.985471 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 276) took 3026ms. Request Metrics: {"tablet-delete.queue_time_us":2778910}
I0619 00:11:56.986183 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 278) took 2998ms. Request Metrics: {"tablet-delete.queue_time_us":2998181}
I0619 00:11:56.986891 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 280) took 2980ms. Request Metrics: {"tablet-delete.queue_time_us":2979900}
I0619 00:11:56.979874 25290 leader_election.cc:379] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:56.987320 25290 leader_election.cc:270] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
I0619 00:11:56.987865 26008 raft_consensus.cc:2732] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.587s (election complete)
I0619 00:11:56.988092 26008 raft_consensus.cc:2539] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 2 FOLLOWER]: Leader election won for term 2
I0619 00:11:56.988382 26008 raft_consensus.cc:593] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [term 2 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:56.988827 26008 consensus_queue.cc:204] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:56.990053 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 54adb588955b45b79de9752f254728ea because the tablet was not found. No further retry: Not found: Tablet not found: 54adb588955b45b79de9752f254728ea
W0619 00:11:56.990559 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 54adb588955b45b79de9752f254728ea because the tablet was not found. No further retry: Not found: Tablet not found: 54adb588955b45b79de9752f254728ea
I0619 00:11:56.990777 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bda4e1ed3e8441ba90f63b78f67c61c5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:56.993692 25084 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 54 ms (attempt = 1)
W0619 00:11:56.994539 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:56.995081 25323 tablet_service.cc:800] Processing DeleteTablet for tablet a284677c4b7f46e18e25edee6024e3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:56.996568 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:56.997289 25290 leader_election.cc:393] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:56.997560 25290 leader_election.cc:270] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:56.975286 25736 raft_consensus.cc:2042] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:56.997807 25736 consensus_peers.cc:456] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:56.998006 26124 raft_consensus.cc:2732] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.853s (election complete)
I0619 00:11:56.998049 25736 consensus_peers.cc:456] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:56.998250 26124 raft_consensus.cc:2784] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:56.998682 25736 raft_consensus.cc:2069] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:56.998970 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) successfully deleted
W0619 00:11:56.999459 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:57.001797 25600 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.002490 26124 raft_consensus.cc:2487] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.004954 26146 raft_consensus.cc:479] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.006306 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 7277051d52dd4be1a6dfa06488c98293 (table default.table_54f94ed54aa24b75bf10f8f57d8d1621 [id=3e2a135384014c7b9626f9a8df3162aa]). Expected version 2 got 1
I0619 00:11:56.984446 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 206) took 2727ms. Request Metrics: {"tablet-delete.queue_time_us":2716224}
I0619 00:11:57.006758 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]). Expected version 1 got 0
I0619 00:11:57.007402 25730 tablet_replica.cc:261] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:57.007692 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(fc7686daae274e0dba87703fe6924805)
I0619 00:11:57.007936 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(fc7686daae274e0dba87703fe6924805)
I0619 00:11:57.007987 25087 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 28 ms (attempt = 1)
I0619 00:11:57.008198 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(fc7686daae274e0dba87703fe6924805)
W0619 00:11:57.008225 25087 catalog_manager.cc:3945] Failed to send 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:57.008519 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(fc7686daae274e0dba87703fe6924805)
I0619 00:11:57.008986 26146 leader_election.cc:243] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.009327 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(fc7686daae274e0dba87703fe6924805)
I0619 00:11:57.009547 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(fc7686daae274e0dba87703fe6924805)
I0619 00:11:57.009766 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(fc7686daae274e0dba87703fe6924805)
I0619 00:11:57.010120 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 161 ms (attempt = 4)
W0619 00:11:57.010422 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
W0619 00:11:57.010807 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
I0619 00:11:57.011162 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19ad911ec6704aef85d9b7f172f08bdc" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:57.011646 25354 raft_consensus.cc:2784] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.011883 25354 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.898s (vote granted)
I0619 00:11:57.012512 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.013303 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.013732 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 281) took 3004ms. Request Metrics: {"tablet-delete.queue_time_us":3003741}
I0619 00:11:57.014322 25741 tablet_replica.cc:261] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:57.014545 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.014787 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.014983 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.015538 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 146 ms (attempt = 4)
W0619 00:11:57.015806 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:57.017241 25354 raft_consensus.cc:2245] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:57.018378 25468 tablet_service.cc:759] Processing CreateTablet for tablet a5c3203e9b6d423f810e1106369da3d7 (table=default.table_9fa9d2f02d024d7592c83d283fbdff04 [id=a78fb9bff11148f2857cfb4e2fda9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:57.018409 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
W0619 00:11:57.018899 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 1dd3f7afe05f49199b18220f9c11ccd2 already in progress: creating tablet
I0619 00:11:57.018893 25947 raft_consensus.cc:457] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.019155 25067 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 25 ms (attempt = 1)
I0619 00:11:57.019151 25947 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.733s (starting election)
W0619 00:11:57.019593 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet e9a221a1d9814ef99a0b4ccaa04d1017 already in progress: creating tablet
I0619 00:11:57.019390 25947 raft_consensus.cc:479] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.019865 25071 catalog_manager.cc:2891] Scheduling retry of e9a221a1d9814ef99a0b4ccaa04d1017 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 39 ms (attempt = 1)
I0619 00:11:57.020256 25727 tablet_bootstrap.cc:427] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
W0619 00:11:57.020298 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 93930ffe285d48d080a8ca4793e871d9 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 93930ffe285d48d080a8ca4793e871d9 already in progress: creating tablet
I0619 00:11:57.020362 25947 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.020509 25071 catalog_manager.cc:2891] Scheduling retry of 93930ffe285d48d080a8ca4793e871d9 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 61 ms (attempt = 1)
I0619 00:11:57.020767 25947 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:57.012964 25558 leader_election.cc:306] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.021400 25558 leader_election.cc:270] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.021893 26151 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.538s (election complete)
I0619 00:11:57.021898 25419 leader_election.cc:379] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.022140 25419 leader_election.cc:270] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:57.022120 26151 raft_consensus.cc:2487] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.019284 25724 tablet_bootstrap.cc:427] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:57.022599 25724 ts_tablet_manager.cc:1045] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.104s	user 0.007s	sys 0.000s
I0619 00:11:57.023450 26100 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.185s (election complete)
I0619 00:11:57.023700 26100 raft_consensus.cc:2539] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:57.023980 26100 raft_consensus.cc:593] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:57.024390 26100 consensus_queue.cc:204] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.027482 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.020709 25727 ts_tablet_manager.cc:1045] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.083s	user 0.002s	sys 0.007s
I0619 00:11:57.031749 25727 raft_consensus.cc:325] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.032335 25727 raft_consensus.cc:351] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.032532 25727 raft_consensus.cc:628] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
W0619 00:11:57.041306 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.015178 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.041859 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.042089 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.042083 26146 leader_election.cc:243] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.042297 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.042464 25741 raft_consensus.cc:2042] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:57.042649 25741 consensus_peers.cc:456] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:57.042867 25741 consensus_peers.cc:456] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
I0619 00:11:57.043226 25741 raft_consensus.cc:2069] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.044818 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 1038 ms (attempt = 7)
W0619 00:11:57.045106 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:57.045248 25741 ts_tablet_manager.cc:1369] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.045435 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:57.049648 25087 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:57.050315 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:57.051770 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.054870 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 1035 ms (attempt = 7)
W0619 00:11:57.055124 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 consensus state has no leader
I0619 00:11:57.055452 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 64 ms (attempt = 2)
W0619 00:11:57.055706 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:57.032963 25727 consensus_queue.cc:227] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.056524 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 54 ms (attempt = 2)
W0619 00:11:57.056820 25071 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:57.001127 25464 ts_tablet_manager.cc:1173] T c1223a8bb4fc4714b962ea5a49903675 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.057329 25462 ts_tablet_manager.cc:1173] T bda4e1ed3e8441ba90f63b78f67c61c5 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:57.058344 25419 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.059957 25467 ts_tablet_manager.cc:1173] T 8140397d408e47129a182fb6255a0a39 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.059960 25087 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:57.060986 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b34043f14f1c4edd97a1d4689ced5b32" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.061378 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:11:57.061741 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f1b566b2b58f4955aa25ae990a4a8a1a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.062043 25084 master_service.cc:523] ReplaceTablet: received request to replace tablet f155a418d0d547f7b775360f1a90ce01 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:57.062682 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a5c3203e9b6d423f810e1106369da3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.067874 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_7ee2ca99e4784301bfb7f6dbbdc2df9e"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.068784 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7ee2ca99e4784301bfb7f6dbbdc2df9e in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.070895 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 3537c79c30a1401e8a78b387f08eff58 with delete_type TABLET_DATA_DELETED (replaced by tablet d0624684b6b04ba3be260b2705dc00e1) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.072719 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet f155a418d0d547f7b775360f1a90ce01
I0619 00:11:57.073012 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f155a418d0d547f7b775360f1a90ce01 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55)
I0619 00:11:57.074926 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f155a418d0d547f7b775360f1a90ce01 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55)
I0619 00:11:57.075300 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f155a418d0d547f7b775360f1a90ce01 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55)
I0619 00:11:57.076381 25084 catalog_manager.cc:4596] ReplaceTablet: tablet f155a418d0d547f7b775360f1a90ce01 deleted and replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55
I0619 00:11:57.079596 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.081403 25463 tablet_service.cc:800] Processing DeleteTablet for tablet a5c3203e9b6d423f810e1106369da3d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.081930 25423 leader_election.cc:379] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.082204 25423 leader_election.cc:270] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:57.082640 25323 tablet_service.cc:800] Processing DeleteTablet for tablet f155a418d0d547f7b775360f1a90ce01 with delete_type TABLET_DATA_DELETED (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.082758 26146 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.908s (election complete)
I0619 00:11:57.082998 26146 raft_consensus.cc:2539] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:57.083205 26146 raft_consensus.cc:457] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.083362 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.083431 26146 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.764s (starting election)
I0619 00:11:57.083688 26146 raft_consensus.cc:2784] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.085108 25463 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.025681 25730 raft_consensus.cc:2042] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:57.023942 25724 raft_consensus.cc:325] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.054823 26151 raft_consensus.cc:457] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.091495 26151 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.296s (starting election)
I0619 00:11:57.091775 26151 raft_consensus.cc:479] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.092741 26151 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.093101 26151 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.088171 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19ad911ec6704aef85d9b7f172f08bdc" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:57.093616 25624 raft_consensus.cc:2784] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.093883 25624 raft_consensus.cc:2732] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.375s (vote granted)
I0619 00:11:57.055702 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.095623 25725 tablet_bootstrap.cc:427] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:57.095973 25725 ts_tablet_manager.cc:1045] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.246s	user 0.006s	sys 0.001s
I0619 00:11:57.096669 25727 ts_tablet_manager.cc:1073] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.066s	user 0.002s	sys 0.000s
I0619 00:11:57.097040 25727 ts_tablet_manager.cc:1028] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:57.086813 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:57.097355 25727 tablet_bootstrap.cc:427] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:57.097362 25623 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.081s (vote granted)
I0619 00:11:57.097579 25623 raft_consensus.cc:2245] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:57.097290 25725 raft_consensus.cc:325] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.097926 25725 raft_consensus.cc:351] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.098110 25725 raft_consensus.cc:628] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:57.099447 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.099860 25624 raft_consensus.cc:2245] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
W0619 00:11:57.101377 25553 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.101658 25736 ts_tablet_manager.cc:1369] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.098768 25725 consensus_queue.cc:227] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.104140 25725 ts_tablet_manager.cc:1073] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I0619 00:11:57.104501 25725 ts_tablet_manager.cc:1028] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:57.105509 25725 tablet_bootstrap.cc:427] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:57.105608 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.106160 25724 raft_consensus.cc:351] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.106338 25423 leader_election.cc:379] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.106361 25724 raft_consensus.cc:628] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:57.106592 25423 leader_election.cc:270] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:57.107034 26092 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.594s (election complete)
I0619 00:11:57.106815 25724 consensus_queue.cc:227] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.107271 26092 raft_consensus.cc:2539] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:57.107493 26092 raft_consensus.cc:457] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.107710 26092 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.867s (starting election)
I0619 00:11:57.107909 26092 raft_consensus.cc:2784] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:57.109508 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.110625 25730 raft_consensus.cc:2069] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.112171 25725 tablet_bootstrap.cc:578] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:57.115101 26092 raft_consensus.cc:479] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:57.090893 25558 leader_election.cc:306] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.116534 25558 leader_election.cc:270] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.116508 26092 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.118072 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:57.118196 25085 catalog_manager.cc:3663] Got report from deleted tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3): Sending delete request for this tablet
I0619 00:11:57.116976 25724 ts_tablet_manager.cc:1073] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.094s	user 0.003s	sys 0.001s
I0619 00:11:57.119144 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
W0619 00:11:57.119443 25419 leader_election.cc:306] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.119927 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 86959e7dda374ba9ac23d3c1fd30d993 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet bae6d308dae54b0394a364d2988ec6b3)
I0619 00:11:57.120857 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_37643042c7dd406790243851c24ade21"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:57.121598 26151 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.022s (election complete)
W0619 00:11:57.121814 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_37643042c7dd406790243851c24ade21 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.121894 26151 raft_consensus.cc:2487] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.119935 25724 ts_tablet_manager.cc:1028] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:57.123464 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 2055 ms (attempt = 8)
W0619 00:11:57.123754 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:57.123978 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:57.124128 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 1031 ms (attempt = 7)
W0619 00:11:57.124373 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:57.124433 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 93 ms (attempt = 3)
I0619 00:11:57.125736 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 0fee3be144c14edab6db58c99e3cda7d with delete_type TABLET_DATA_DELETED (replaced by tablet 6755b36ed905430e8724d43920c9c0d8) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.126790 26092 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.128711 26140 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=1} inserts{seen=10 ignored=0} mutations{seen=1 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0619 00:11:57.130815 26140 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Bootstrap complete.
W0619 00:11:57.132639 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.122102 26146 raft_consensus.cc:479] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.135754 25724 tablet_bootstrap.cc:427] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:57.136958 25470 tablet_service.cc:800] Processing DeleteTablet for tablet a4c653a255984d3bac88df63362b9334 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.137703 26092 consensus_queue.cc:963] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:57.137958 25724 tablet_bootstrap.cc:578] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:57.125764 25085 catalog_manager.cc:3663] Got report from deleted tablet 54373917567a479d93dc8a3fd006d2fe (table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]) (Table deleted at 2018-06-19 00:11:51 PDT): Sending delete request for this tablet
W0619 00:11:57.138718 25071 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:57.136032 25080 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:11:57.139175 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:11:57.139458 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:57.126354 25067 catalog_manager.cc:2891] Scheduling retry of e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC with a delay of 139 ms (attempt = 4)
W0619 00:11:57.140100 25067 catalog_manager.cc:2910] Async tablet task e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e5051fc388d74ad39753c43b0ad25402 consensus state has no leader
I0619 00:11:57.140537 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 108 ms (attempt = 3)
W0619 00:11:57.140833 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:57.141141 25727 tablet_bootstrap.cc:578] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:57.145117 25470 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.146751 25085 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:57.146874 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_782346c257df4294bd8390a894d670f8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:57.147403 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
W0619 00:11:57.147873 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_782346c257df4294bd8390a894d670f8 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.138496 26146 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.148423 25468 ts_tablet_manager.cc:1173] T a5c3203e9b6d423f810e1106369da3d7 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.150131 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:57.152555 25419 leader_election.cc:306] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.157577 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 54373917567a479d93dc8a3fd006d2fe with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:51 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.152936 26146 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.158498 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.160245 26176 tablet.cc:1200] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
W0619 00:11:57.164068 25290 consensus_peers.cc:434] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet 7f60ea5c5def47ee846d0186bbc90c18. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:57.165400 25068 catalog_manager.cc:2891] Scheduling retry of 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC with a delay of 286 ms (attempt = 5)
W0619 00:11:57.165683 25068 catalog_manager.cc:2910] Async tablet task 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 19ad911ec6704aef85d9b7f172f08bdc consensus state has no leader
I0619 00:11:57.166821 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.166911 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_27e7515742a84639bd10967f26b8d93f" } new_table_name: "default.table_c9352e571d1b4c3da9d5ef85ec9e9139" alter_external_catalogs: true
I0619 00:11:57.117782 25730 ts_tablet_manager.cc:1369] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.138859 26151 raft_consensus.cc:457] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.167824 26151 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.228s (starting election)
I0619 00:11:57.168253 25087 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:57.168097 26151 raft_consensus.cc:479] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.168905 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:57.169116 26151 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.169502 26151 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.164779 25600 tablet_service.cc:800] Processing DeleteTablet for tablet f155a418d0d547f7b775360f1a90ce01 with delete_type TABLET_DATA_DELETED (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.165659 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.166407 25598 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:57.171880 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.172202 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 556 ms (attempt = 6)
I0619 00:11:57.173168 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.179138 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.179965 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 3722b203a516466489c74ef6013eacae with delete_type TABLET_DATA_DELETED (replaced by tablet fe83202d55bc40e8ae193b9f6f7e9adf) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.180332 25470 tablet_service.cc:759] Processing CreateTablet for tablet 33783498947e4e038e876f7e409eea30 (table=default.table_07a87b17722646a38c8a8cd224140386 [id=e1814d17f14e45f4b45f1372005e6f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.180969 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 33783498947e4e038e876f7e409eea30. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.182145 26175 tablet.cc:1200] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:57.183130 25085 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 23 ms (attempt = 1)
W0619 00:11:57.183409 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
W0619 00:11:57.181118 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:57.184418 25085 master_service.cc:523] ReplaceTablet: received request to replace tablet cbd67634f5704165b5bb948501697f0c from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:57.167207 25558 leader_election.cc:306] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.185197 25558 leader_election.cc:270] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.185873 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 293 ms (attempt = 5)
I0619 00:11:57.185976 26151 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.795s (election complete)
W0619 00:11:57.186203 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:57.186203 26151 raft_consensus.cc:2487] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.187125 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_d0723d646ccd440c850471e54eb7078d" }
I0619 00:11:57.188359 25079 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_9a8c9964bcff4effb2b624ef5bfdef1c" }
I0619 00:11:57.189749 25727 tablet_bootstrap.cc:427] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:57.190140 25727 ts_tablet_manager.cc:1045] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.093s	user 0.004s	sys 0.002s
I0619 00:11:57.190894 25470 ts_tablet_manager.cc:1173] T 33783498947e4e038e876f7e409eea30 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.192867 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_402a7a51052c418d8617889c96683206"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.193804 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_402a7a51052c418d8617889c96683206 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:57.193805 25290 consensus_peers.cc:434] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 7f60ea5c5def47ee846d0186bbc90c18. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18. Retrying in the next heartbeat period. Already tried 3 times.
I0619 00:11:57.198418 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.199131 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.199836 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.202117 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.202555 25487 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.218s (vote granted)
I0619 00:11:57.202787 25487 raft_consensus.cc:2245] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
W0619 00:11:57.204421 25423 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
I0619 00:11:57.204740 25423 leader_election.cc:270] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:57.205116 25423 raft_consensus.cc:2442] T edfdedbd4eca4b62be24b1649f820d0e P 10bda0b4856f4441b5aed24cdc4764a8: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:11:57.207511 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 58 ms (attempt = 2)
W0619 00:11:57.207836 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:11:57.208393 25725 tablet_bootstrap.cc:427] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:57.208766 25725 ts_tablet_manager.cc:1045] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.103s	user 0.006s	sys 0.001s
I0619 00:11:57.209915 25468 tablet_service.cc:800] Processing DeleteTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.210647 25725 raft_consensus.cc:325] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.211779 25736 ts_tablet_manager.cc:1382] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W0619 00:11:57.211980 25553 leader_election.cc:306] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.212057 25736 log.cc:986] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/3537c79c30a1401e8a78b387f08eff58
I0619 00:11:57.211549 25725 raft_consensus.cc:351] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.213543 25725 raft_consensus.cc:628] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:57.214107 25725 consensus_queue.cc:227] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:57.216132 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:11:57.218049 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.223130 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3031f8f5176e4479af3b3f3d51ebdb06" } new_table_name: "default.table_d74a1686828b4ee4b21231219c545181" alter_external_catalogs: true
I0619 00:11:57.228055 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.227313 25080 catalog_manager.cc:3786] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:57.228391 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]). Expected version 1 got 0
I0619 00:11:57.228823 25080 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]). Expected version 1 got 0
I0619 00:11:57.229158 25081 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_54f94ed54aa24b75bf10f8f57d8d1621 [id=3e2a135384014c7b9626f9a8df3162aa] alter complete (version 2)
I0619 00:11:57.191612 25727 raft_consensus.cc:325] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.232998 25724 tablet_bootstrap.cc:427] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:57.233074 25727 raft_consensus.cc:351] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.233351 25724 ts_tablet_manager.cc:1045] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.098s	user 0.004s	sys 0.000s
I0619 00:11:57.233377 25727 raft_consensus.cc:628] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:57.234004 25727 consensus_queue.cc:227] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.235574 25086 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC with a delay of 44 ms (attempt = 1)
W0619 00:11:57.235877 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 has no consensus state
I0619 00:11:57.236552 25724 raft_consensus.cc:325] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.237210 25724 raft_consensus.cc:351] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.237417 25724 raft_consensus.cc:628] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:57.205116 25470 tablet_service.cc:759] Processing CreateTablet for tablet 992f255e14544af3b8a89cd295944f29 (table=default.table_d2a13e99b1964be7a97603e011c9cb6b [id=08e4e7f0b7864a91ad75b900b2a89186]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.242372 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 992f255e14544af3b8a89cd295944f29. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.243022 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 2654c7e3655d46b0bac8251218b538e3
I0619 00:11:57.243324 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2654c7e3655d46b0bac8251218b538e3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.244159 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2654c7e3655d46b0bac8251218b538e3 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.244521 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2654c7e3655d46b0bac8251218b538e3 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.244668 25727 ts_tablet_manager.cc:1073] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.054s	user 0.004s	sys 0.000s
I0619 00:11:57.245014 25727 ts_tablet_manager.cc:1028] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:57.245496 25727 tablet_bootstrap.cc:427] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:57.247879 25727 tablet_bootstrap.cc:578] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:57.249488 25080 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 44 ms (attempt = 1)
W0619 00:11:57.249781 25080 catalog_manager.cc:3945] Failed to send 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:57.252032 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 169 ms (attempt = 4)
W0619 00:11:57.252305 25071 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:57.252897 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 162 ms (attempt = 4)
I0619 00:11:57.254082 25468 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:57.253144 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:57.256794 25079 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 4feb648bb30247b49a4abd49da95ea7d
I0619 00:11:57.257087 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4feb648bb30247b49a4abd49da95ea7d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.257452 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4feb648bb30247b49a4abd49da95ea7d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.257844 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4feb648bb30247b49a4abd49da95ea7d on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.259016 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 2654c7e3655d46b0bac8251218b538e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.259919 25323 tablet_service.cc:759] Processing CreateTablet for tablet 7d0a40f08b344572917bf54ecc22cd20 (table=default.table_35bd36a899a9473f80f5e06739f90c1b [id=e75cd72c58d142d698aeb990dd81222e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.260695 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7d0a40f08b344572917bf54ecc22cd20. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.261404 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.262367 25468 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.263645 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.212745 25736 ts_tablet_manager.cc:1403] T 3537c79c30a1401e8a78b387f08eff58 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:57.266515 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 276) took 3154ms. Request Metrics: {"tablet-delete.queue_time_us":2855340}
I0619 00:11:57.267261 25736 tablet_replica.cc:261] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:57.267215 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 89 ms (attempt = 3)
W0619 00:11:57.267494 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:11:57.267550 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.267781 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.267971 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.268328 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.268677 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.269222 25336 tablet_service.cc:759] Processing CreateTablet for tablet 894de151fa3b4a4fbc962a5cafbccd3a (table=default.table_947e41e56412414da7590c66fe5fc49a [id=9fbdae7ea5534755bcf89cf6a87f5748]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.269402 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 4feb648bb30247b49a4abd49da95ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.269845 25326 tablet_service.cc:759] Processing CreateTablet for tablet 6755b36ed905430e8724d43920c9c0d8 (table=default.table_c71886d2a0ac4d869a6806cf551bba7e [id=e4d5113658b04af49f2b7d2e67cf71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.270354 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet cbd67634f5704165b5bb948501697f0c
I0619 00:11:57.270679 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cbd67634f5704165b5bb948501697f0c on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 507f7ed270af42ce8f404d55b5cae468)
I0619 00:11:57.270769 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6755b36ed905430e8724d43920c9c0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.271081 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cbd67634f5704165b5bb948501697f0c on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 507f7ed270af42ce8f404d55b5cae468)
I0619 00:11:57.271473 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cbd67634f5704165b5bb948501697f0c on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 507f7ed270af42ce8f404d55b5cae468)
I0619 00:11:57.271814 25085 catalog_manager.cc:4596] ReplaceTablet: tablet cbd67634f5704165b5bb948501697f0c deleted and replaced by tablet 507f7ed270af42ce8f404d55b5cae468
I0619 00:11:57.254082 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 2654c7e3655d46b0bac8251218b538e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.254665 25601 tablet_service.cc:759] Processing CreateTablet for tablet 7d0a40f08b344572917bf54ecc22cd20 (table=default.table_35bd36a899a9473f80f5e06739f90c1b [id=e75cd72c58d142d698aeb990dd81222e]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:57.275866 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.276217 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7d0a40f08b344572917bf54ecc22cd20. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.264618 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:57.276504 25624 raft_consensus.cc:2784] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.276787 25624 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.310s (vote granted)
I0619 00:11:57.265302 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 4feb648bb30247b49a4abd49da95ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.255264 25600 tablet_service.cc:759] Processing CreateTablet for tablet 894de151fa3b4a4fbc962a5cafbccd3a (table=default.table_947e41e56412414da7590c66fe5fc49a [id=9fbdae7ea5534755bcf89cf6a87f5748]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.280088 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 894de151fa3b4a4fbc962a5cafbccd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.281401 25624 raft_consensus.cc:2245] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:57.266780 25558 leader_election.cc:379] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.282094 25558 leader_election.cc:270] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:57.282868 26151 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.563s (election complete)
I0619 00:11:57.283092 26151 raft_consensus.cc:2539] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:57.283275 26151 raft_consensus.cc:457] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.283452 26151 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.767s (starting election)
I0619 00:11:57.283685 26151 raft_consensus.cc:2784] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.285879 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 894de151fa3b4a4fbc962a5cafbccd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.286358 25067 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC with a delay of 64 ms (attempt = 2)
W0619 00:11:57.286660 25067 catalog_manager.cc:2910] Async tablet task 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 consensus state has no leader
I0619 00:11:57.286988 25067 catalog_manager.cc:2891] Scheduling retry of e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC with a delay of 271 ms (attempt = 5)
W0619 00:11:57.287230 25067 catalog_manager.cc:2910] Async tablet task e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e5051fc388d74ad39753c43b0ad25402 consensus state has no leader
I0619 00:11:57.287560 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 2087 ms (attempt = 8)
W0619 00:11:57.287832 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:57.288770 25331 tablet_service.cc:800] Processing DeleteTablet for tablet cbd67634f5704165b5bb948501697f0c with delete_type TABLET_DATA_DELETED (replaced by tablet 507f7ed270af42ce8f404d55b5cae468) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.288462 26151 raft_consensus.cc:479] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.289522 26151 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.289947 26151 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.290328 26008 raft_consensus.cc:457] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.290565 26008 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.027s (starting election)
I0619 00:11:57.290824 26008 raft_consensus.cc:479] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.291772 26008 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.291857 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:57.292501 25600 ts_tablet_manager.cc:1173] T 894de151fa3b4a4fbc962a5cafbccd3a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.292712 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
W0619 00:11:57.293423 25553 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.294103 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 65 ms (attempt = 2)
W0619 00:11:57.294381 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:57.294827 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.295230 26008 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.295604 25724 consensus_queue.cc:227] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.298506 25724 ts_tablet_manager.cc:1073] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.065s	user 0.004s	sys 0.000s
I0619 00:11:57.298818 25724 ts_tablet_manager.cc:1028] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:57.299160 25724 tablet_bootstrap.cc:427] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:57.299238 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 4144 ms (attempt = 9)
W0619 00:11:57.299579 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.301165 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]). Expected version 2 got 0
I0619 00:11:57.301436 25724 tablet_bootstrap.cc:578] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:57.302031 25085 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 24 ms (attempt = 1)
W0619 00:11:57.302328 25085 catalog_manager.cc:3945] Failed to send 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.307843 25350 raft_consensus.cc:1081] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:57.308981 25323 ts_tablet_manager.cc:1173] T 7d0a40f08b344572917bf54ecc22cd20 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.309432 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 3537c79c30a1401e8a78b387f08eff58 (table default.table_da957178ddc241ed8e414551ec468a12 [id=0cd55331c72b42bfb2583be0283a06ef]) successfully deleted
I0619 00:11:57.309262 26176 tablet.cc:1195] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 2 got alter request for version 2
I0619 00:11:57.310732 25085 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]). Expected version 2 got 0
I0619 00:11:57.311482 25085 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 39 ms (attempt = 1)
W0619 00:11:57.311806 25085 catalog_manager.cc:3945] Failed to send 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.316005 25601 ts_tablet_manager.cc:1173] T 7d0a40f08b344572917bf54ecc22cd20 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.316738 25600 tablet_service.cc:800] Processing DeleteTablet for tablet cbd67634f5704165b5bb948501697f0c with delete_type TABLET_DATA_DELETED (replaced by tablet 507f7ed270af42ce8f404d55b5cae468) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.317713 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 1034 ms (attempt = 7)
W0619 00:11:57.318001 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:57.321851 25326 ts_tablet_manager.cc:1173] T 6755b36ed905430e8724d43920c9c0d8 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.322655 25600 tablet_service.cc:759] Processing CreateTablet for tablet 6755b36ed905430e8724d43920c9c0d8 (table=default.table_c71886d2a0ac4d869a6806cf551bba7e [id=e4d5113658b04af49f2b7d2e67cf71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.323215 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6755b36ed905430e8724d43920c9c0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.284744 26145 raft_consensus.cc:457] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.305114 25468 tablet_service.cc:800] Processing DeleteTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:57.306308 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.327364 26145 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.532s (starting election)
I0619 00:11:57.307075 25463 tablet_service.cc:800] Processing DeleteTablet for tablet c1223a8bb4fc4714b962ea5a49903675 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.310405 26100 consensus_queue.cc:963] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:57.327636 26145 raft_consensus.cc:479] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.323928 25947 raft_consensus.cc:457] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.328403 25947 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.954s (starting election)
I0619 00:11:57.328708 26145 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:57.328721 25558 leader_election.cc:306] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.319272 25741 ts_tablet_manager.cc:1382] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:57.329162 26145 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.329082 25947 raft_consensus.cc:479] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.329455 25741 log.cc:986] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a4c653a255984d3bac88df63362b9334
I0619 00:11:57.308569 25462 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.330003 25947 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.330366 25947 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.330562 25741 ts_tablet_manager.cc:1403] T a4c653a255984d3bac88df63362b9334 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:57.331471 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.331845 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(a4c653a255984d3bac88df63362b9334)
I0619 00:11:57.332062 25736 raft_consensus.cc:2042] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:57.332365 25736 raft_consensus.cc:2069] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.332352 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.334918 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:57.335191 25419 leader_election.cc:306] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.335989 25736 ts_tablet_manager.cc:1369] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.333415 25730 ts_tablet_manager.cc:1382] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:57.334877 25602 tablet_service.cc:759] Processing CreateTablet for tablet fe83202d55bc40e8ae193b9f6f7e9adf (table=default.table_ec6a5ef017d74c7eb1777f1bf058a3da [id=a17737f9d2924cc09f372527fb91188b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.337013 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fe83202d55bc40e8ae193b9f6f7e9adf. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.337473 25467 tablet_service.cc:759] Processing CreateTablet for tablet e5051fc388d74ad39753c43b0ad25402 (table=default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:57.337728 25419 leader_election.cc:306] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.338083 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5051fc388d74ad39753c43b0ad25402. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.338894 26175 tablet.cc:1195] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
I0619 00:11:57.336115 25598 tablet_service.cc:759] Processing CreateTablet for tablet ed3b0dace2e2475aa466039465b44a84 (table=default.table_25bd5ae167b74dea974c7eabdd1cd7b0 [id=84d4a84fd4d243eea76c6308866adb82]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.340070 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ed3b0dace2e2475aa466039465b44a84. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.342245 25470 ts_tablet_manager.cc:1173] T 992f255e14544af3b8a89cd295944f29 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.345002 25725 ts_tablet_manager.cc:1073] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.136s	user 0.005s	sys 0.000s
I0619 00:11:57.349508 25067 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 67 ms (attempt = 2)
W0619 00:11:57.349907 25067 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.350864 26008 raft_consensus.cc:457] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.351121 26008 raft_consensus.cc:2732] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.780s (starting election)
I0619 00:11:57.351358 26008 raft_consensus.cc:479] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.352710 26008 leader_election.cc:243] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.353068 26008 leader_election.cc:243] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.354522 25598 ts_tablet_manager.cc:1173] T ed3b0dace2e2475aa466039465b44a84 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.355315 25336 ts_tablet_manager.cc:1173] T 894de151fa3b4a4fbc962a5cafbccd3a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.355549 25730 log.cc:986] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/fc7686daae274e0dba87703fe6924805
I0619 00:11:57.336982 25601 tablet_service.cc:759] Processing CreateTablet for tablet 92fc5418174246bba92169339f264d62 (table=default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.357777 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 167 ms (attempt = 4)
I0619 00:11:57.356175 25558 leader_election.cc:270] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:57.358058 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:11:57.358707 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 65 ms (attempt = 2)
W0619 00:11:57.359186 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.361513 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 108 ms (attempt = 3)
W0619 00:11:57.361814 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:57.363274 25079 catalog_manager.cc:3663] Got report from deleted tablet 3e7724f967ae41d7b523055d1a4f58db (table default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:57.363973 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:11:57.349359 25423 consensus_peers.cc:434] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 075076aaa16e4ea8a596a7a8c52f103e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 075076aaa16e4ea8a596a7a8c52f103e. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:57.367568 25067 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC with a delay of 110 ms (attempt = 3)
W0619 00:11:57.367915 25067 catalog_manager.cc:2910] Async tablet task 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 consensus state has no leader
I0619 00:11:57.352530 25467 ts_tablet_manager.cc:1173] T e5051fc388d74ad39753c43b0ad25402 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.369277 25725 ts_tablet_manager.cc:1028] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:57.369612 25725 tablet_bootstrap.cc:427] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:57.358358 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 92fc5418174246bba92169339f264d62. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.369858 25079 catalog_manager.cc:3663] Got report from deleted tablet 3e7724f967ae41d7b523055d1a4f58db (table default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:57.370503 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:57.372272 25725 tablet_bootstrap.cc:578] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:57.372836 25290 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:57.373538 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.376128 25602 ts_tablet_manager.cc:1173] T fe83202d55bc40e8ae193b9f6f7e9adf P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.376682 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ac07448d0aa64bf7b1c84f5011880149"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:57.377146 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_f18ea472f11c4efa81d1e00b99a5a22a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.377677 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ac07448d0aa64bf7b1c84f5011880149 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.377634 26151 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.749s (election complete)
I0619 00:11:57.377890 26151 raft_consensus.cc:2487] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.378427 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:57.373978 25290 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a284677c4b7f46e18e25edee6024e3eb
W0619 00:11:57.379591 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 51026.5us
W0619 00:11:57.380201 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f18ea472f11c4efa81d1e00b99a5a22a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.380246 25290 leader_election.cc:270] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.369606 25730 ts_tablet_manager.cc:1403] T fc7686daae274e0dba87703fe6924805 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:57.381747 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 213) took 3025ms. Request Metrics: {"tablet-delete.queue_time_us":2650863}
I0619 00:11:57.382557 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) successfully deleted
I0619 00:11:57.382647 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 212) took 3026ms. Request Metrics: {"tablet-delete.queue_time_us":3002349}
W0619 00:11:57.384094 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
I0619 00:11:57.384349 25336 tablet_service.cc:759] Processing CreateTablet for tablet 92fc5418174246bba92169339f264d62 (table=default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.389935 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3b5e020c7a0148b48bc211eb1dc414ec" } new_table_name: "default.table_c4520a4e6b374312a9f1801b938645c0" alter_external_catalogs: true
W0619 00:11:57.390666 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.391054 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_17ebb06ba3d041478a1d93d0cb9e3964"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.391973 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_17ebb06ba3d041478a1d93d0cb9e3964 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.379348 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 282) took 3368ms. Request Metrics: {"tablet-delete.queue_time_us":3003257}
I0619 00:11:57.398253 25326 tablet_service.cc:759] Processing CreateTablet for tablet fe83202d55bc40e8ae193b9f6f7e9adf (table=default.table_ec6a5ef017d74c7eb1777f1bf058a3da [id=a17737f9d2924cc09f372527fb91188b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.398679 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:57.398910 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fe83202d55bc40e8ae193b9f6f7e9adf. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.399113 25624 raft_consensus.cc:2784] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.399355 25624 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.396s (vote granted)
I0619 00:11:57.403447 25724 tablet_bootstrap.cc:427] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:57.403797 25724 ts_tablet_manager.cc:1045] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.105s	user 0.006s	sys 0.001s
I0619 00:11:57.403975 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 92fc5418174246bba92169339f264d62. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.404173 25600 ts_tablet_manager.cc:1173] T 6755b36ed905430e8724d43920c9c0d8 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.407820 26008 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.731s (election complete)
I0619 00:11:57.408068 26008 raft_consensus.cc:2487] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.384053 26175 tablet.cc:1195] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 2 got alter request for version 2
I0619 00:11:57.395781 25462 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.397016 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.412128 25080 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:57.412804 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:57.397770 25470 tablet_service.cc:800] Processing DeleteTablet for tablet f155a418d0d547f7b775360f1a90ce01 with delete_type TABLET_DATA_DELETED (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.421321 25423 leader_election.cc:379] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.421320 25067 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 76 ms (attempt = 3)
I0619 00:11:57.421573 25423 leader_election.cc:270] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:11:57.421685 25067 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.422045 25947 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.697s (election complete)
I0619 00:11:57.422093 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 261 ms (attempt = 5)
I0619 00:11:57.422282 25947 raft_consensus.cc:2539] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
W0619 00:11:57.422329 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:57.422489 25947 raft_consensus.cc:457] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.422726 25947 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.071s (starting election)
I0619 00:11:57.422926 25947 raft_consensus.cc:2784] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.423082 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 299 ms (attempt = 5)
W0619 00:11:57.423347 25071 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:57.423880 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:57.424813 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 86959e7dda374ba9ac23d3c1fd30d993 with delete_type TABLET_DATA_DELETED (replaced by tablet bae6d308dae54b0394a364d2988ec6b3) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.425596 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 106 ms (attempt = 3)
W0619 00:11:57.425889 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.428695 25947 raft_consensus.cc:479] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.429808 25947 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.430495 25947 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.431876 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_359e958d80b547aaa28ef250b6a3c56c"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.432749 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_359e958d80b547aaa28ef250b6a3c56c in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.434505 25084 catalog_manager.cc:2891] Scheduling retry of c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC with a delay of 16 ms (attempt = 1)
W0619 00:11:57.434785 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c244dec62a5b4ee492a9a110b656a561 consensus state has no leader
I0619 00:11:57.429977 25487 raft_consensus.cc:2175] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:57.440909 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:57.441203 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.441876 25423 leader_election.cc:379] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.443269 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_bd5ecdaeb9784fe696613bd457e08878" } new_table_name: "default.table_b5b52789c2e347ccb0587438d7002a6b" alter_external_catalogs: true
I0619 00:11:57.443490 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 2654c7e3655d46b0bac8251218b538e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:57.443706 25419 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.444808 25470 tablet_service.cc:759] Processing CreateTablet for tablet 7d0a40f08b344572917bf54ecc22cd20 (table=default.table_35bd36a899a9473f80f5e06739f90c1b [id=e75cd72c58d142d698aeb990dd81222e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.445418 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7d0a40f08b344572917bf54ecc22cd20. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.447387 25086 catalog_manager.cc:3663] Got report from deleted tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:57.448045 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:57.451334 25069 catalog_manager.cc:2891] Scheduling retry of c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC with a delay of 52 ms (attempt = 2)
W0619 00:11:57.451606 25069 catalog_manager.cc:2910] Async tablet task c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c244dec62a5b4ee492a9a110b656a561 consensus state has no leader
I0619 00:11:57.454401 25084 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 55 ms (attempt = 1)
W0619 00:11:57.454722 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 has no consensus state
I0619 00:11:57.455960 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_718ed0cda54b4ab8b71539f7f044d28f" }
I0619 00:11:57.405051 25724 raft_consensus.cc:325] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.456874 25724 raft_consensus.cc:351] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.457082 25724 raft_consensus.cc:628] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:57.457535 25724 consensus_queue.cc:227] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.458202 25725 tablet_bootstrap.cc:427] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:57.458544 25725 ts_tablet_manager.cc:1045] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.089s	user 0.005s	sys 0.001s
I0619 00:11:57.459188 25724 ts_tablet_manager.cc:1073] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.055s	user 0.003s	sys 0.001s
I0619 00:11:57.459805 25725 raft_consensus.cc:325] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.460384 25725 raft_consensus.cc:351] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.447079 26187 tablet.cc:1200] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 2
I0619 00:11:57.406050 25624 raft_consensus.cc:2245] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:57.462900 25736 ts_tablet_manager.cc:1382] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:57.463227 25736 log.cc:986] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a4c653a255984d3bac88df63362b9334
I0619 00:11:57.463714 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.463860 25736 ts_tablet_manager.cc:1403] T a4c653a255984d3bac88df63362b9334 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:57.465795 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_ff6a2c70c1d94d7a814d423d400f6c33" }
I0619 00:11:57.466248 25336 ts_tablet_manager.cc:1173] T 92fc5418174246bba92169339f264d62 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:57.466817 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 92845.6us
I0619 00:11:57.468015 25323 tablet_service.cc:759] Processing CreateTablet for tablet ed3b0dace2e2475aa466039465b44a84 (table=default.table_25bd5ae167b74dea974c7eabdd1cd7b0 [id=84d4a84fd4d243eea76c6308866adb82]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.468608 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ed3b0dace2e2475aa466039465b44a84. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.469662 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:57.470098 25623 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.741s (vote granted)
I0619 00:11:57.470283 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 128 ms (attempt = 4)
I0619 00:11:57.470315 25623 raft_consensus.cc:2245] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
W0619 00:11:57.470535 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:57.396311 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 283) took 3298ms. Request Metrics: {"tablet-delete.queue_time_us":3291691}
I0619 00:11:57.471402 25741 tablet_replica.cc:261] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
W0619 00:11:57.471500 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.470913 26176 tablet.cc:1195] T 7277051d52dd4be1a6dfa06488c98293 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 2 got alter request for version 2
W0619 00:11:57.474874 25423 consensus_peers.cc:434] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet 54adb588955b45b79de9752f254728ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54adb588955b45b79de9752f254728ea. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:57.476147 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.476605 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) successfully deleted
W0619 00:11:57.478113 25290 leader_election.cc:306] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 54373917567a479d93dc8a3fd006d2fe
I0619 00:11:57.479487 25067 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC with a delay of 177 ms (attempt = 4)
W0619 00:11:57.479785 25067 catalog_manager.cc:2910] Async tablet task 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 consensus state has no leader
I0619 00:11:57.390049 26140 raft_consensus.cc:325] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:57.480206 26140 raft_consensus.cc:628] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a016cc9f274bc1979068306ec6d15e, State: Initialized, Role: FOLLOWER
I0619 00:11:57.460569 25725 raft_consensus.cc:628] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:57.481075 26140 consensus_queue.cc:227] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:57.481755 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 518 ms (attempt = 6)
W0619 00:11:57.482015 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:57.481021 25725 consensus_queue.cc:227] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.482820 25558 leader_election.cc:393] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:57.483121 25558 leader_election.cc:270] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:11:57.483784 26151 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.011s (election complete)
I0619 00:11:57.484022 26151 raft_consensus.cc:2487] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.485605 25462 tablet_service.cc:759] Processing CreateTablet for tablet 894de151fa3b4a4fbc962a5cafbccd3a (table=default.table_947e41e56412414da7590c66fe5fc49a [id=9fbdae7ea5534755bcf89cf6a87f5748]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.486239 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 894de151fa3b4a4fbc962a5cafbccd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.486732 25947 raft_consensus.cc:457] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.486968 25947 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.055s (starting election)
I0619 00:11:57.487197 25947 raft_consensus.cc:479] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.488131 25947 leader_election.cc:243] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.488986 25947 leader_election.cc:243] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.489500 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e5e691e6d374cd4bba1f18d8c821723" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.408978 25730 tablet_replica.cc:261] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
W0619 00:11:57.490754 25419 leader_election.cc:306] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.490914 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:57.491205 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:57.491523 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:57.491786 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:57.492050 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:57.492038 25470 ts_tablet_manager.cc:1173] T 7d0a40f08b344572917bf54ecc22cd20 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.492278 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:57.492471 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:57.492667 25730 raft_consensus.cc:2042] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:57.492931 25730 raft_consensus.cc:2069] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.494215 25730 ts_tablet_manager.cc:1369] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.494734 25724 ts_tablet_manager.cc:1028] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:57.495110 25724 tablet_bootstrap.cc:427] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:57.495486 26151 raft_consensus.cc:457] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.495740 26151 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.103s (starting election)
I0619 00:11:57.495973 26151 raft_consensus.cc:479] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.496901 26151 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.496728 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 281) took 3214ms. Request Metrics: {"mutex_wait_us":4292,"tablet-delete.queue_time_us":2983982}
I0619 00:11:57.497684 26151 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.497608 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 283) took 3176ms. Request Metrics: {"tablet-delete.queue_time_us":3165359}
W0619 00:11:57.498515 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.498576 25323 ts_tablet_manager.cc:1173] T ed3b0dace2e2475aa466039465b44a84 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.499714 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.500319 25423 leader_election.cc:379] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.501706 25085 catalog_manager.cc:3663] Got report from deleted tablet 6040e369e0fb445fb993d12dd883039b (table default.table_673676f1e1e94d76afe2c9b06d701801 [id=980403da36b04fad89427ba22b96d098]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:57.502429 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:57.503764 25601 ts_tablet_manager.cc:1173] T 92fc5418174246bba92169339f264d62 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.504106 25069 catalog_manager.cc:2891] Scheduling retry of c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC with a delay of 101 ms (attempt = 3)
W0619 00:11:57.504391 25069 catalog_manager.cc:2910] Async tablet task c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c244dec62a5b4ee492a9a110b656a561 consensus state has no leader
I0619 00:11:57.505903 25067 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 170 ms (attempt = 4)
W0619 00:11:57.506171 25067 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.506139 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:57.506494 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
W0619 00:11:57.510628 25553 leader_election.cc:306] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.511318 26151 raft_consensus.cc:457] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.511555 26151 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.550s (starting election)
I0619 00:11:57.512096 25727 tablet_bootstrap.cc:427] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:57.511811 26151 raft_consensus.cc:479] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.512430 25727 ts_tablet_manager.cc:1045] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.267s	user 0.003s	sys 0.006s
I0619 00:11:57.512773 26151 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.512809 25326 ts_tablet_manager.cc:1173] T fe83202d55bc40e8ae193b9f6f7e9adf P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.513141 26151 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.514044 25067 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 73 ms (attempt = 2)
W0619 00:11:57.514219 25067 catalog_manager.cc:2910] Async tablet task 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 has no consensus state
I0619 00:11:57.514721 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.498308 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 285) took 3126ms. Request Metrics: {"tablet-delete.queue_time_us":3124392}
I0619 00:11:57.500748 25725 ts_tablet_manager.cc:1073] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.000s
I0619 00:11:57.508085 25423 leader_election.cc:270] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:57.520248 25947 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.054s (election complete)
I0619 00:11:57.520480 25947 raft_consensus.cc:2539] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:57.520784 25947 raft_consensus.cc:593] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
W0619 00:11:57.521160 25553 leader_election.cc:306] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.521175 25947 consensus_queue.cc:204] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.522528 25724 tablet_bootstrap.cc:578] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:57.528214 25351 raft_consensus.cc:1081] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:57.528910 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 270 ms (attempt = 5)
W0619 00:11:57.529194 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:11:57.529862 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:57.530338 25624 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.095s (vote granted)
I0619 00:11:57.530582 25624 raft_consensus.cc:2245] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:57.530724 26092 consensus_queue.cc:963] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:57.535279 25071 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC with a delay of 2072 ms (attempt = 8)
W0619 00:11:57.535547 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 consensus state has no leader
I0619 00:11:57.535894 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 136 ms (attempt = 4)
W0619 00:11:57.536128 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.540102 25601 tablet_service.cc:759] Processing CreateTablet for tablet 420ea76207564b68aafd1833814cc595 (table=default.table_402a7a51052c418d8617889c96683206 [id=df3a613e287049fa95612862e96c6888]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.540745 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 420ea76207564b68aafd1833814cc595. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.540973 25086 master_service.cc:523] ReplaceTablet: received request to replace tablet b50379774e4d4fdfb823ab2dc919db30 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:57.543992 25600 tablet_service.cc:759] Processing CreateTablet for tablet 60a3b5eaeae44965929232031deda216 (table=default.table_782346c257df4294bd8390a894d670f8 [id=4d8390b19cb14623976c859b8ab7e056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.544574 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60a3b5eaeae44965929232031deda216. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.545120 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:57.545379 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:57.545562 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:57.545773 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:57.545997 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:57.546190 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:57.546363 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(7f60ea5c5def47ee846d0186bbc90c18)
I0619 00:11:57.546522 25741 raft_consensus.cc:2042] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:57.546727 25741 consensus_peers.cc:456] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:57.546924 25741 consensus_peers.cc:456] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
I0619 00:11:57.547458 25741 raft_consensus.cc:2069] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.548794 25068 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 4100 ms (attempt = 9)
I0619 00:11:57.548871 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 4c92d341c4d94e4987524a056e703310
W0619 00:11:57.549077 25068 catalog_manager.cc:2910] Async tablet task f155a418d0d547f7b775360f1a90ce01 (table default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f155a418d0d547f7b775360f1a90ce01 consensus state has no leader
I0619 00:11:57.549175 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c92d341c4d94e4987524a056e703310 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.549576 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c92d341c4d94e4987524a056e703310 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.549965 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4c92d341c4d94e4987524a056e703310 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.551764 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 4c92d341c4d94e4987524a056e703310 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.556234 25727 raft_consensus.cc:325] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.556872 25727 raft_consensus.cc:351] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.557065 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet bda4e1ed3e8441ba90f63b78f67c61c5
I0619 00:11:57.557057 25727 raft_consensus.cc:628] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:57.557337 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bda4e1ed3e8441ba90f63b78f67c61c5 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.557735 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bda4e1ed3e8441ba90f63b78f67c61c5 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.557487 25727 consensus_queue.cc:227] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.558102 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bda4e1ed3e8441ba90f63b78f67c61c5 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.559677 25598 tablet_service.cc:800] Processing DeleteTablet for tablet bda4e1ed3e8441ba90f63b78f67c61c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.559969 25067 catalog_manager.cc:2891] Scheduling retry of e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC with a delay of 521 ms (attempt = 6)
I0619 00:11:57.560207 25741 ts_tablet_manager.cc:1369] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:57.560253 25067 catalog_manager.cc:2910] Async tablet task e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e5051fc388d74ad39753c43b0ad25402 consensus state has no leader
I0619 00:11:57.562700 25598 tablet_service.cc:759] Processing CreateTablet for tablet 13265ab1253f44d6b3ffe61fa0141be1 (table=default.table_37643042c7dd406790243851c24ade21 [id=6d61f318c5e74320b6c45eeecc382d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.563284 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 13265ab1253f44d6b3ffe61fa0141be1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.564385 25724 tablet_bootstrap.cc:427] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:57.564779 25724 ts_tablet_manager.cc:1045] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.070s	user 0.007s	sys 0.004s
W0619 00:11:57.565572 25290 leader_election.cc:306] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 54373917567a479d93dc8a3fd006d2fe
I0619 00:11:57.565871 25290 leader_election.cc:270] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.566253 25724 raft_consensus.cc:325] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.566906 25724 raft_consensus.cc:351] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.567116 25724 raft_consensus.cc:628] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:57.567682 25724 consensus_queue.cc:227] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.568985 25727 ts_tablet_manager.cc:1073] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.056s	user 0.005s	sys 0.000s
I0619 00:11:57.570873 25600 ts_tablet_manager.cc:1173] T 60a3b5eaeae44965929232031deda216 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.572744 25724 ts_tablet_manager.cc:1073] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I0619 00:11:57.573150 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet a4c653a255984d3bac88df63362b9334 (table default.table_c771d75362c64504aed46e585f8fbdf3 [id=94bd8944b43d4eb19150de690e983915]) successfully deleted
I0619 00:11:57.533498 26188 sys_catalog.cc:337] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:57.573767 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 4feb648bb30247b49a4abd49da95ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:57.574183 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 1dd3f7afe05f49199b18220f9c11ccd2 already in progress: creating tablet
I0619 00:11:57.574400 25071 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 24 ms (attempt = 1)
I0619 00:11:57.574503 25727 ts_tablet_manager.cc:1028] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:57.574829 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 775f0b6b309c49278066a5e7cf623af2 because the tablet was not found. No further retry: Not found: Tablet not found: 775f0b6b309c49278066a5e7cf623af2
I0619 00:11:57.576288 25727 tablet_bootstrap.cc:427] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:57.507855 25462 ts_tablet_manager.cc:1173] T 894de151fa3b4a4fbc962a5cafbccd3a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.577234 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 286) took 3197ms. Request Metrics: {"tablet-delete.queue_time_us":3139599}
I0619 00:11:57.577921 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 291) took 3117ms. Request Metrics: {"tablet-delete.queue_time_us":3092635}
I0619 00:11:57.578454 25736 tablet_replica.cc:261] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:57.578714 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.578925 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.579035 25598 ts_tablet_manager.cc:1173] T 13265ab1253f44d6b3ffe61fa0141be1 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.579124 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.579330 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.579378 26188 sys_catalog.cc:340] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:57.579561 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.579799 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.580024 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.579984 25602 tablet_service.cc:759] Processing CreateTablet for tablet 82f1b4e87c3249a6af8a1f27f10a639e (table=default.table_7ee2ca99e4784301bfb7f6dbbdc2df9e [id=2bfabb7d0cf44dc5a112e214dbbc21b4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.580193 25736 raft_consensus.cc:2042] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:57.580384 25736 consensus_peers.cc:456] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:57.580554 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 82f1b4e87c3249a6af8a1f27f10a639e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.580575 25736 consensus_peers.cc:456] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:57.581010 25736 raft_consensus.cc:2069] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.582911 25736 ts_tablet_manager.cc:1369] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.584339 25599 tablet_service.cc:759] Processing CreateTablet for tablet 507f7ed270af42ce8f404d55b5cae468 (table=default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.584928 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 507f7ed270af42ce8f404d55b5cae468. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:57.587239 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.587671 25725 ts_tablet_manager.cc:1028] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:57.588076 25725 tablet_bootstrap.cc:427] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:57.587992 25603 tablet_service.cc:759] Processing CreateTablet for tablet 18dbdb33a25e487ab87ecb85af31fd55 (table=default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.588534 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:57.589519 25489 raft_consensus.cc:1081] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 2 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I0619 00:11:57.589772 25489 pending_rounds.cc:76] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Aborting all transactions after (but not including) 0
I0619 00:11:57.589923 25489 pending_rounds.cc:97] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Aborting uncommitted NO_OP operation due to leader change: 1.1
I0619 00:11:57.590093 25489 raft_consensus.cc:2622] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I0619 00:11:57.590576 25940 raft_consensus.cc:457] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.590849 25940 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.216s (starting election)
I0619 00:11:57.591099 25940 raft_consensus.cc:479] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.592089 25940 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.592489 25940 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.588553 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 18dbdb33a25e487ab87ecb85af31fd55. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.599910 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 270 ms (attempt = 5)
W0619 00:11:57.600188 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:57.602038 25602 ts_tablet_manager.cc:1173] T 82f1b4e87c3249a6af8a1f27f10a639e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.602980 25725 tablet_bootstrap.cc:578] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:57.606897 25069 catalog_manager.cc:2891] Scheduling retry of c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC with a delay of 132 ms (attempt = 4)
W0619 00:11:57.607192 25069 catalog_manager.cc:2910] Async tablet task c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c244dec62a5b4ee492a9a110b656a561 consensus state has no leader
I0619 00:11:57.608808 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:57.609586 25067 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 95 ms (attempt = 3)
W0619 00:11:57.609881 25067 catalog_manager.cc:2910] Async tablet task 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 consensus state has no leader
I0619 00:11:57.610844 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.611328 25940 raft_consensus.cc:2732] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.995s (election complete)
I0619 00:11:57.611554 25940 raft_consensus.cc:2487] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:57.612131 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 46535.8us
W0619 00:11:57.612928 25069 catalog_manager.cc:2910] Async tablet task 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb648bb30247b49a4abd49da95ea7d consensus state has no leader
I0619 00:11:57.613206 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.614519 25727 tablet_bootstrap.cc:578] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:57.616833 25336 tablet_service.cc:759] Processing CreateTablet for tablet 60a3b5eaeae44965929232031deda216 (table=default.table_782346c257df4294bd8390a894d670f8 [id=4d8390b19cb14623976c859b8ab7e056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.616992 25622 raft_consensus.cc:1081] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 2 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I0619 00:11:57.617398 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60a3b5eaeae44965929232031deda216. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.617534 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.618010 25489 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.525s (vote granted)
I0619 00:11:57.618232 25489 raft_consensus.cc:2245] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
I0619 00:11:57.619323 25470 tablet_service.cc:800] Processing DeleteTablet for tablet cbd67634f5704165b5bb948501697f0c with delete_type TABLET_DATA_DELETED (replaced by tablet 507f7ed270af42ce8f404d55b5cae468) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.619689 25326 tablet_service.cc:759] Processing CreateTablet for tablet 420ea76207564b68aafd1833814cc595 (table=default.table_402a7a51052c418d8617889c96683206 [id=df3a613e287049fa95612862e96c6888]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.620256 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 420ea76207564b68aafd1833814cc595. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.622527 25730 ts_tablet_manager.cc:1382] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:57.622953 25730 log.cc:986] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/d6c3c7f783cb43d59b46502261aeced3
I0619 00:11:57.624419 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 4c92d341c4d94e4987524a056e703310 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.625350 25323 tablet_service.cc:759] Processing CreateTablet for tablet 13265ab1253f44d6b3ffe61fa0141be1 (table=default.table_37643042c7dd406790243851c24ade21 [id=6d61f318c5e74320b6c45eeecc382d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.625918 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 13265ab1253f44d6b3ffe61fa0141be1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.626085 25603 ts_tablet_manager.cc:1173] T 18dbdb33a25e487ab87ecb85af31fd55 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.627398 25599 ts_tablet_manager.cc:1173] T 507f7ed270af42ce8f404d55b5cae468 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.629829 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_45234f5ba11946f390d40924c779fe3e"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.630749 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_45234f5ba11946f390d40924c779fe3e in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.630890 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "981d862d1bab4b14bb10e6fc2174e1cc" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.631326 25489 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.644s (vote granted)
I0619 00:11:57.631554 25489 raft_consensus.cc:2245] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:57.631660 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_3676937cf5b04949a7a870594522a5fa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.632488 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3676937cf5b04949a7a870594522a5fa in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.632788 25470 tablet_service.cc:759] Processing CreateTablet for tablet 6755b36ed905430e8724d43920c9c0d8 (table=default.table_c71886d2a0ac4d869a6806cf551bba7e [id=e4d5113658b04af49f2b7d2e67cf71e9]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.633376 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6755b36ed905430e8724d43920c9c0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.633172 25624 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e5e691e6d374cd4bba1f18d8c821723" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:57.633612 25624 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.023s (vote granted)
I0619 00:11:57.633853 25624 raft_consensus.cc:2245] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:57.637307 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_06ca188d31e6404585c64d19afe3dc8a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.638182 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_06ca188d31e6404585c64d19afe3dc8a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.638612 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_359e958d80b547aaa28ef250b6a3c56c" } new_table_name: "default.table_493f801c4cb6466ab9ae71ff0b25c30e" alter_external_catalogs: true
I0619 00:11:57.643489 25470 ts_tablet_manager.cc:1173] T 6755b36ed905430e8724d43920c9c0d8 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.647136 25724 ts_tablet_manager.cc:1028] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:57.648131 25724 tablet_bootstrap.cc:427] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:57.652537 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_5f896be5c2ab4c36bbf4ce4bcb10e345"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:57.653414 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5f896be5c2ab4c36bbf4ce4bcb10e345 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.655071 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.658027 25067 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC with a delay of 275 ms (attempt = 5)
W0619 00:11:57.658290 25067 catalog_manager.cc:2910] Async tablet task 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 consensus state has no leader
I0619 00:11:57.659837 25078 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_fd67adcd1cd54957a8ffbe59f5f3e833" } new_table_name: "default.table_6486a9f255374893bfb9f02f620c80da" alter_external_catalogs: true
I0619 00:11:57.660085 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:57.661533 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7c57c5ca55f24621b66197a3698504bd because the tablet was not found. No further retry: Not found: Tablet not found: 7c57c5ca55f24621b66197a3698504bd
W0619 00:11:57.662356 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
I0619 00:11:57.662792 25462 tablet_service.cc:759] Processing CreateTablet for tablet fe83202d55bc40e8ae193b9f6f7e9adf (table=default.table_ec6a5ef017d74c7eb1777f1bf058a3da [id=a17737f9d2924cc09f372527fb91188b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.663604 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fe83202d55bc40e8ae193b9f6f7e9adf. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.654808 25940 raft_consensus.cc:457] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.665931 25940 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.846s (starting election)
I0619 00:11:57.663288 25323 ts_tablet_manager.cc:1173] T 13265ab1253f44d6b3ffe61fa0141be1 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.666357 25940 raft_consensus.cc:479] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.667043 25470 tablet_service.cc:759] Processing CreateTablet for tablet 92fc5418174246bba92169339f264d62 (table=default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.667649 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 92fc5418174246bba92169339f264d62. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.667532 25940 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.668285 25940 leader_election.cc:243] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.667124 25730 ts_tablet_manager.cc:1403] T d6c3c7f783cb43d59b46502261aeced3 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:11:57.625393 25290 leader_election.cc:306] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 981d862d1bab4b14bb10e6fc2174e1cc
W0619 00:11:57.669486 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 44153.9us
I0619 00:11:57.670814 25940 consensus_queue.cc:963] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I0619 00:11:57.672407 26151 raft_consensus.cc:457] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.672562 25727 tablet_bootstrap.cc:427] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:57.672678 26151 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.024s (starting election)
I0619 00:11:57.672914 25727 ts_tablet_manager.cc:1045] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.097s	user 0.004s	sys 0.001s
I0619 00:11:57.672916 26151 raft_consensus.cc:479] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.673889 26151 leader_election.cc:243] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.673765 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 272 ms (attempt = 5)
W0619 00:11:57.674193 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.674265 26151 leader_election.cc:243] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.675057 25724 tablet_bootstrap.cc:578] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:57.675371 26175 tablet.cc:1200] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:57.676300 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41e052fdd92b4c3d9803fa43433e60d5" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:57.680233 25067 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 274 ms (attempt = 5)
W0619 00:11:57.680518 25067 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
W0619 00:11:57.680764 25553 leader_election.cc:306] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.686271 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 522 ms (attempt = 6)
W0619 00:11:57.686542 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:57.685376 25558 leader_election.cc:379] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.687090 25323 tablet_service.cc:800] Processing DeleteTablet for tablet bda4e1ed3e8441ba90f63b78f67c61c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.687826 25068 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 2097 ms (attempt = 8)
I0619 00:11:57.687379 25558 leader_election.cc:270] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:11:57.688097 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:57.690146 26151 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.000s (election complete)
I0619 00:11:57.690374 26151 raft_consensus.cc:2539] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:57.690564 26151 raft_consensus.cc:457] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.690769 26151 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.057s (starting election)
I0619 00:11:57.690954 26151 raft_consensus.cc:2784] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.695364 26151 raft_consensus.cc:479] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.697129 26151 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.697793 26151 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.702379 26176 tablet.cc:1200] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:57.703230 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:57.703841 26175 tablet.cc:1195] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
I0619 00:11:57.707803 25725 tablet_bootstrap.cc:427] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:57.708851 25067 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 141 ms (attempt = 4)
W0619 00:11:57.709126 25067 catalog_manager.cc:2910] Async tablet task 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 consensus state has no leader
I0619 00:11:57.709161 25725 ts_tablet_manager.cc:1045] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.121s	user 0.006s	sys 0.000s
W0619 00:11:57.710352 25553 leader_election.cc:306] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:57.712290 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.715121 26176 tablet.cc:1195] T 19ad911ec6704aef85d9b7f172f08bdc P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
I0619 00:11:57.716532 25736 ts_tablet_manager.cc:1382] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I0619 00:11:57.716852 25736 log.cc:986] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/ee4c4769f75141f9b2ef864ab8a8d9e0
I0619 00:11:57.717576 25736 ts_tablet_manager.cc:1403] T ee4c4769f75141f9b2ef864ab8a8d9e0 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
W0619 00:11:57.717180 25558 leader_election.cc:306] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.717885 25558 leader_election.cc:270] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.719007 25323 tablet_service.cc:759] Processing CreateTablet for tablet 82f1b4e87c3249a6af8a1f27f10a639e (table=default.table_7ee2ca99e4784301bfb7f6dbbdc2df9e [id=2bfabb7d0cf44dc5a112e214dbbc21b4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.719892 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet b50379774e4d4fdfb823ab2dc919db30
I0619 00:11:57.720115 25741 ts_tablet_manager.cc:1382] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:57.720206 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b50379774e4d4fdfb823ab2dc919db30 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb)
I0619 00:11:57.720443 25741 log.cc:986] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:57.720638 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b50379774e4d4fdfb823ab2dc919db30 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb)
I0619 00:11:57.721025 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b50379774e4d4fdfb823ab2dc919db30 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb)
I0619 00:11:57.721303 25084 catalog_manager.cc:3663] Got report from deleted tablet e0070a115e8540cda5f999394deac616 (table default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:57.721560 25741 ts_tablet_manager.cc:1403] T 7f60ea5c5def47ee846d0186bbc90c18 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:57.721637 25084 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:57.721323 25086 catalog_manager.cc:4596] ReplaceTablet: tablet b50379774e4d4fdfb823ab2dc919db30 deleted and replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb
I0619 00:11:57.725577 25083 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]). Expected version 1 got 0
I0619 00:11:57.726372 25467 tablet_service.cc:759] Processing CreateTablet for tablet ed3b0dace2e2475aa466039465b44a84 (table=default.table_25bd5ae167b74dea974c7eabdd1cd7b0 [id=84d4a84fd4d243eea76c6308866adb82]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.727016 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ed3b0dace2e2475aa466039465b44a84. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.728456 25463 tablet_service.cc:800] Processing DeleteTablet for tablet ee4c4769f75141f9b2ef864ab8a8d9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.729323 25423 leader_election.cc:379] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.729619 25423 leader_election.cc:270] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:57.730114 25947 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.618s (election complete)
I0619 00:11:57.730350 25947 raft_consensus.cc:2539] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:57.730641 25947 raft_consensus.cc:593] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:57.731175 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 518 ms (attempt = 6)
W0619 00:11:57.731423 25071 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:57.731107 25947 consensus_queue.cc:204] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.735069 25599 tablet_service.cc:800] Processing DeleteTablet for tablet b50379774e4d4fdfb823ab2dc919db30 with delete_type TABLET_DATA_DELETED (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.736377 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41e052fdd92b4c3d9803fa43433e60d5" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:11:57.737820 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.738165 25940 consensus_queue.cc:963] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:57.739706 25069 catalog_manager.cc:2891] Scheduling retry of c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC with a delay of 269 ms (attempt = 5)
W0619 00:11:57.740036 25069 catalog_manager.cc:2910] Async tablet task c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c244dec62a5b4ee492a9a110b656a561 consensus state has no leader
I0619 00:11:57.741832 25727 raft_consensus.cc:325] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.742458 25727 raft_consensus.cc:351] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.742683 25727 raft_consensus.cc:628] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:57.743099 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 1042 ms (attempt = 7)
W0619 00:11:57.743388 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:57.748092 25078 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 65 ms (attempt = 1)
W0619 00:11:57.749075 25078 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
I0619 00:11:57.750200 25085 catalog_manager.cc:3663] Got report from deleted tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:57.750516 25085 catalog_manager.cc:3663] Got report from deleted tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) (Table deleted at 2018-06-19 00:11:50 PDT): Sending delete request for this tablet
I0619 00:11:57.750821 25085 catalog_manager.cc:3663] Got report from deleted tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:57.751790 26151 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.571s (election complete)
I0619 00:11:57.752777 25725 raft_consensus.cc:325] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.753391 25725 raft_consensus.cc:351] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.753568 25725 raft_consensus.cc:628] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:57.754454 25725 consensus_queue.cc:227] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.753706 25601 ts_tablet_manager.cc:1173] T 420ea76207564b68aafd1833814cc595 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.755782 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:57.756326 25622 raft_consensus.cc:2175] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:57.755432 26151 raft_consensus.cc:2487] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.757252 25725 ts_tablet_manager.cc:1073] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.047s	user 0.004s	sys 0.000s
I0619 00:11:57.757570 25725 ts_tablet_manager.cc:1028] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:57.757922 25725 tablet_bootstrap.cc:427] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:57.758193 25467 ts_tablet_manager.cc:1173] T ed3b0dace2e2475aa466039465b44a84 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.759006 25462 ts_tablet_manager.cc:1173] T fe83202d55bc40e8ae193b9f6f7e9adf P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.719825 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 294) took 3176ms. Request Metrics: {"tablet-delete.queue_time_us":3031592}
I0619 00:11:57.759469 25727 consensus_queue.cc:227] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.744982 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 82f1b4e87c3249a6af8a1f27f10a639e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.760524 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 295) took 3194ms. Request Metrics: {"tablet-delete.queue_time_us":3192660}
I0619 00:11:57.761188 25736 tablet_replica.cc:261] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:57.761456 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:57.761652 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:57.761602 25624 raft_consensus.cc:1081] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:57.761857 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:57.762034 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:57.762239 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:57.762439 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:57.762701 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:57.762905 25736 raft_consensus.cc:2042] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:57.763108 25736 consensus_peers.cc:456] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:57.763308 25736 consensus_peers.cc:456] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:57.759292 25079 master_service.cc:523] ReplaceTablet: received request to replace tablet 86959e7dda374ba9ac23d3c1fd30d993 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:57.763671 25736 raft_consensus.cc:2069] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.753015 25290 leader_election.cc:379] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.765023 25290 leader_election.cc:270] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:57.766430 25078 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_8efbcf7cfbc0497fb5d8fb885d82e4ca" }
I0619 00:11:57.767225 25463 tablet_service.cc:759] Processing CreateTablet for tablet 420ea76207564b68aafd1833814cc595 (table=default.table_402a7a51052c418d8617889c96683206 [id=df3a613e287049fa95612862e96c6888]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.767307 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_9751fcfca6454dd99468df3ffb363e82" }
I0619 00:11:57.767647 25336 ts_tablet_manager.cc:1173] T 60a3b5eaeae44965929232031deda216 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.768023 25423 leader_election.cc:379] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.768321 25423 leader_election.cc:270] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:57.749363 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:57.768815 26092 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.650s (election complete)
I0619 00:11:57.769047 26092 raft_consensus.cc:2539] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:57.769121 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:57.769254 26092 raft_consensus.cc:457] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.769451 26092 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.919s (starting election)
I0619 00:11:57.769646 26092 raft_consensus.cc:2784] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.770560 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 214) took 3373ms. Request Metrics: {"spinlock_wait_cycles":123056896,"tablet-delete.queue_time_us":2983203}
I0619 00:11:57.771307 25730 tablet_replica.cc:261] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:57.773496 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(b34043f14f1c4edd97a1d4689ced5b32)
I0619 00:11:57.773936 25326 ts_tablet_manager.cc:1173] T 420ea76207564b68aafd1833814cc595 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.774209 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(b34043f14f1c4edd97a1d4689ced5b32)
I0619 00:11:57.744941 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 284) took 3626ms. Request Metrics: {"mutex_wait_us":64621,"tablet-delete.queue_time_us":3333285}
I0619 00:11:57.774381 25940 raft_consensus.cc:457] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.774658 25940 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.090s (starting election)
I0619 00:11:57.774770 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(b34043f14f1c4edd97a1d4689ced5b32)
I0619 00:11:57.774976 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(b34043f14f1c4edd97a1d4689ced5b32)
I0619 00:11:57.776784 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(b34043f14f1c4edd97a1d4689ced5b32)
I0619 00:11:57.777014 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(b34043f14f1c4edd97a1d4689ced5b32)
I0619 00:11:57.777215 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(b34043f14f1c4edd97a1d4689ced5b32)
I0619 00:11:57.777393 25730 raft_consensus.cc:2042] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:57.777673 25730 raft_consensus.cc:2069] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.774935 25940 raft_consensus.cc:479] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.778735 25940 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.775687 26124 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.625s (election complete)
I0619 00:11:57.779024 26124 raft_consensus.cc:2539] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:57.779240 26124 raft_consensus.cc:457] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.779114 25940 leader_election.cc:243] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.779459 26124 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.813s (starting election)
I0619 00:11:57.779706 26124 raft_consensus.cc:2784] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.780972 25724 tablet_bootstrap.cc:427] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:57.781319 25724 ts_tablet_manager.cc:1045] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.133s	user 0.006s	sys 0.002s
I0619 00:11:57.783834 25462 tablet_service.cc:759] Processing CreateTablet for tablet 60a3b5eaeae44965929232031deda216 (table=default.table_782346c257df4294bd8390a894d670f8 [id=4d8390b19cb14623976c859b8ab7e056]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.773154 25470 ts_tablet_manager.cc:1173] T 92fc5418174246bba92169339f264d62 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.784276 26124 raft_consensus.cc:479] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.784467 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c244dec62a5b4ee492a9a110b656a561" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.785459 26124 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.785873 26124 leader_election.cc:243] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.785835 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:57.786269 25490 raft_consensus.cc:2784] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.785861 25724 raft_consensus.cc:325] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.788406 25331 tablet_service.cc:759] Processing CreateTablet for tablet 18dbdb33a25e487ab87ecb85af31fd55 (table=default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:57.789361 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.790840 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 18dbdb33a25e487ab87ecb85af31fd55. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.793867 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 286) took 3670ms. Request Metrics: {"tablet-delete.queue_time_us":3638078}
I0619 00:11:57.794507 25741 tablet_replica.cc:261] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:57.794797 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.796599 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet e5051fc388d74ad39753c43b0ad25402
I0619 00:11:57.796897 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5051fc388d74ad39753c43b0ad25402 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.797013 25087 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 36 ms (attempt = 1)
W0619 00:11:57.797300 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 has no consensus state
I0619 00:11:57.775429 25727 ts_tablet_manager.cc:1073] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.102s	user 0.003s	sys 0.000s
I0619 00:11:57.797386 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5051fc388d74ad39753c43b0ad25402 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.797446 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.797670 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.797782 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5051fc388d74ad39753c43b0ad25402 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.797914 25336 tablet_service.cc:759] Processing CreateTablet for tablet 507f7ed270af42ce8f404d55b5cae468 (table=default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.799589 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 539 ms (attempt = 6)
W0619 00:11:57.799928 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:11:57.798504 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 507f7ed270af42ce8f404d55b5cae468. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.800528 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.800828 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.801045 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.801239 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(ee4c4769f75141f9b2ef864ab8a8d9e0)
I0619 00:11:57.801326 25330 tablet_service.cc:800] Processing DeleteTablet for tablet b50379774e4d4fdfb823ab2dc919db30 with delete_type TABLET_DATA_DELETED (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.801424 25741 raft_consensus.cc:2042] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:57.801700 25741 raft_consensus.cc:2069] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.802191 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 7f60ea5c5def47ee846d0186bbc90c18 (table default.table_62e2a91242c24687b8c59b83aa396c02 [id=9df75da386114e098e64d8388744a316]) successfully deleted
I0619 00:11:57.802482 25730 ts_tablet_manager.cc:1369] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.803364 25741 ts_tablet_manager.cc:1369] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.803761 25727 ts_tablet_manager.cc:1028] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:57.804129 25727 tablet_bootstrap.cc:427] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:57.805915 25078 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet f24f9ab30ad04f39b5ac27e557606ab1
I0619 00:11:57.806226 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f24f9ab30ad04f39b5ac27e557606ab1 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.786474 25724 raft_consensus.cc:351] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.806628 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f24f9ab30ad04f39b5ac27e557606ab1 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.806748 25724 raft_consensus.cc:628] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:57.807060 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f24f9ab30ad04f39b5ac27e557606ab1 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:57.807389 25330 tablet_service.cc:800] Processing DeleteTablet for tablet e5051fc388d74ad39753c43b0ad25402 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.807642 25085 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81] alter complete (version 1)
I0619 00:11:57.808523 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:57.808915 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:11:57.809281 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:57.807759 25724 consensus_queue.cc:227] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.813226 25423 leader_election.cc:379] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.813346 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) successfully deleted
I0619 00:11:57.813520 25423 leader_election.cc:270] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:57.813401 25085 catalog_manager.cc:3663] Got report from deleted tablet e0070a115e8540cda5f999394deac616 (table default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
W0619 00:11:57.813872 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet e60ecc51c92c461889b360dde1dc5cf2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet e60ecc51c92c461889b360dde1dc5cf2 already in progress: creating tablet
I0619 00:11:57.814091 25071 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 27 ms (attempt = 1)
I0619 00:11:57.814309 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:57.814743 25068 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 61 ms (attempt = 2)
W0619 00:11:57.815016 25068 catalog_manager.cc:2910] Async tablet task 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
I0619 00:11:57.815347 25331 ts_tablet_manager.cc:1173] T 18dbdb33a25e487ab87ecb85af31fd55 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.817066 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 420ea76207564b68aafd1833814cc595. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.819095 25724 ts_tablet_manager.cc:1073] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.034s	user 0.004s	sys 0.000s
I0619 00:11:57.819419 25724 ts_tablet_manager.cc:1028] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:57.821518 26092 raft_consensus.cc:479] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.823127 26092 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.823537 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60a3b5eaeae44965929232031deda216. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.830159 25323 ts_tablet_manager.cc:1173] T 82f1b4e87c3249a6af8a1f27f10a639e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.786495 25490 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.848s (vote granted)
I0619 00:11:57.831370 26092 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.831987 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 4c92d341c4d94e4987524a056e703310 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.832728 25727 tablet_bootstrap.cc:578] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:57.833603 25068 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 60 ms (attempt = 2)
W0619 00:11:57.833847 25068 catalog_manager.cc:2910] Async tablet task 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 has no consensus state
I0619 00:11:57.834543 25330 tablet_service.cc:800] Processing DeleteTablet for tablet f24f9ab30ad04f39b5ac27e557606ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.835535 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.836036 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.836307 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 7f60ea5c5def47ee846d0186bbc90c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:50 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:57.836809 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
I0619 00:11:57.837018 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.837594 25330 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.838574 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:57.840524 25419 leader_election.cc:306] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.840590 26124 raft_consensus.cc:457] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.840843 26124 raft_consensus.cc:2732] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.825s (starting election)
I0619 00:11:57.841084 26124 raft_consensus.cc:479] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.841461 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.842054 26124 leader_election.cc:243] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:57.843515 25470 tablet_service.cc:759] Processing CreateTablet for tablet 13265ab1253f44d6b3ffe61fa0141be1 (table=default.table_37643042c7dd406790243851c24ade21 [id=6d61f318c5e74320b6c45eeecc382d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.843715 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:57.844154 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 13265ab1253f44d6b3ffe61fa0141be1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.845018 25488 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:57.845633 25488 raft_consensus.cc:2206] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:57.848587 26124 leader_election.cc:243] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.848928 25467 tablet_service.cc:800] Processing DeleteTablet for tablet bda4e1ed3e8441ba90f63b78f67c61c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.854058 25067 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 268 ms (attempt = 5)
W0619 00:11:57.854315 25067 catalog_manager.cc:2910] Async tablet task 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 consensus state has no leader
I0619 00:11:57.856200 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "981d862d1bab4b14bb10e6fc2174e1cc" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:57.857084 25486 raft_consensus.cc:2784] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:57.857331 25486 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.613s (vote granted)
I0619 00:11:57.858283 25468 tablet_service.cc:759] Processing CreateTablet for tablet 507f7ed270af42ce8f404d55b5cae468 (table=default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.858938 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 507f7ed270af42ce8f404d55b5cae468. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.857311 25467 tablet_service.cc:759] Processing CreateTablet for tablet 82f1b4e87c3249a6af8a1f27f10a639e (table=default.table_7ee2ca99e4784301bfb7f6dbbdc2df9e [id=2bfabb7d0cf44dc5a112e214dbbc21b4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.860944 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 82f1b4e87c3249a6af8a1f27f10a639e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.871037 25069 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 548 ms (attempt = 6)
I0619 00:11:57.871021 25736 ts_tablet_manager.cc:1369] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:57.871318 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
W0619 00:11:57.857520 25423 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:11:57.875226 25423 leader_election.cc:270] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:11:57.875752 26146 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.082s (election complete)
I0619 00:11:57.875982 26146 raft_consensus.cc:2487] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:11:57.876452 25068 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 81 ms (attempt = 3)
W0619 00:11:57.876742 25068 catalog_manager.cc:2910] Async tablet task 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
I0619 00:11:57.876646 25464 tablet_service.cc:759] Processing CreateTablet for tablet 18dbdb33a25e487ab87ecb85af31fd55 (table=default.table_fd67adcd1cd54957a8ffbe59f5f3e833 [id=65652a104e3943e69b26a960c2bb1e62]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:57.876855 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.877302 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 18dbdb33a25e487ab87ecb85af31fd55. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:57.877813 25469 tablet_service.cc:800] Processing DeleteTablet for tablet b50379774e4d4fdfb823ab2dc919db30 with delete_type TABLET_DATA_DELETED (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.877929 25489 raft_consensus.cc:2206] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:57.881090 25423 leader_election.cc:379] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.881350 25423 leader_election.cc:270] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:57.882441 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.883106 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 54adb588955b45b79de9752f254728ea with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.884280 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e5051fc388d74ad39753c43b0ad25402 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.884470 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 2053 ms (attempt = 8)
W0619 00:11:57.884811 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:57.884932 25469 tablet_service.cc:800] Processing DeleteTablet for tablet f24f9ab30ad04f39b5ac27e557606ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.876080 25290 leader_election.cc:393] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:57.885526 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:57.885977 25947 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.521s (election complete)
I0619 00:11:57.886234 25947 raft_consensus.cc:2539] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:57.886440 25947 raft_consensus.cc:457] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:57.886637 25947 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.703s (starting election)
I0619 00:11:57.886862 25947 raft_consensus.cc:2784] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:57.888221 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.889494 25741 ts_tablet_manager.cc:1382] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I0619 00:11:57.896404 25068 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 93 ms (attempt = 3)
I0619 00:11:57.894595 25727 tablet_bootstrap.cc:427] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
W0619 00:11:57.896721 25068 catalog_manager.cc:2910] Async tablet task 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 has no consensus state
I0619 00:11:57.896823 25741 log.cc:986] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/ee4c4769f75141f9b2ef864ab8a8d9e0
I0619 00:11:57.896996 25727 ts_tablet_manager.cc:1045] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.093s	user 0.004s	sys 0.004s
I0619 00:11:57.897725 25085 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:57.898396 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:57.898504 25725 tablet_bootstrap.cc:578] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:57.899879 25290 leader_election.cc:393] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:57.903121 25741 ts_tablet_manager.cc:1403] T ee4c4769f75141f9b2ef864ab8a8d9e0 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:57.903782 25490 raft_consensus.cc:2245] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 1.
I0619 00:11:57.905203 25463 ts_tablet_manager.cc:1173] T 420ea76207564b68aafd1833814cc595 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.905606 26092 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.900s (election complete)
I0619 00:11:57.905855 26092 raft_consensus.cc:2502] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I0619 00:11:57.911463 25486 raft_consensus.cc:2245] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:57.912547 25467 ts_tablet_manager.cc:1173] T 82f1b4e87c3249a6af8a1f27f10a639e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.915067 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]) successfully deleted
I0619 00:11:57.917826 25468 ts_tablet_manager.cc:1173] T 507f7ed270af42ce8f404d55b5cae468 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:57.918258 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 200772us
I0619 00:11:57.918406 25947 raft_consensus.cc:479] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.919935 25470 ts_tablet_manager.cc:1173] T 13265ab1253f44d6b3ffe61fa0141be1 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:57.921358 25558 leader_election.cc:306] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.921604 25558 leader_election.cc:270] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.922473 25464 ts_tablet_manager.cc:1173] T 18dbdb33a25e487ab87ecb85af31fd55 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:57.923399 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC with a delay of 2081 ms (attempt = 8)
W0619 00:11:57.923666 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:57.924166 25947 leader_election.cc:243] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.924190 25599 tablet_service.cc:800] Processing DeleteTablet for tablet f24f9ab30ad04f39b5ac27e557606ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.925190 25947 leader_election.cc:243] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.925720 25601 tablet_service.cc:800] Processing DeleteTablet for tablet e5051fc388d74ad39753c43b0ad25402 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:57.928736 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e5e691e6d374cd4bba1f18d8c821723" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:57.930233 25419 leader_election.cc:306] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:57.930989 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.932154 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 289) took 3762ms. Request Metrics: {"tablet-delete.queue_time_us":3624031}
I0619 00:11:57.934381 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:57.934794 25067 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC with a delay of 541 ms (attempt = 6)
W0619 00:11:57.935072 25067 catalog_manager.cc:2910] Async tablet task 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 consensus state has no leader
I0619 00:11:57.938045 25724 tablet_bootstrap.cc:427] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:57.938868 25741 tablet_replica.cc:261] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:57.939116 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:57.939324 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:57.939512 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:57.939740 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:57.939970 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:57.940177 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:57.940369 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(a52cbcbe35374d518bca1e0fc24d88d4)
I0619 00:11:57.940549 25741 raft_consensus.cc:2042] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:57.940838 25741 raft_consensus.cc:2069] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:57.942081 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 2087 ms (attempt = 8)
W0619 00:11:57.942345 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:11:57.943792 25724 tablet_bootstrap.cc:578] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:57.945287 25078 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_4b709074f52242faa541fac2119805ed"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:57.945444 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ea3c06e3d9864c2b81bd1aa252b3281a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:57.945853 25462 ts_tablet_manager.cc:1173] T 60a3b5eaeae44965929232031deda216 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:57.946303 25078 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4b709074f52242faa541fac2119805ed in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:57.946328 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ea3c06e3d9864c2b81bd1aa252b3281a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:57.947981 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:57.898396 25727 raft_consensus.cc:325] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.948137 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 549 ms (attempt = 6)
W0619 00:11:57.948396 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.949365 25727 raft_consensus.cc:351] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.949311 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_73944be4ee474c9e9e3f654a4ed45b55"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:57.948968 25947 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:57.950168 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_73944be4ee474c9e9e3f654a4ed45b55 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:57.950117 25947 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.150s (starting election)
I0619 00:11:57.950435 25947 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.951483 25947 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:57.954147 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_b6fb03b2b1ed49ab94d4e017c1ba1408" } new_table_name: "default.table_3c040b90a5ff40c180e4679dee37fd1f" alter_external_catalogs: true
I0619 00:11:57.956028 25067 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 551 ms (attempt = 6)
W0619 00:11:57.956286 25067 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:57.953802 25947 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:57.958756 25724 tablet_bootstrap.cc:427] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:57.959090 25724 ts_tablet_manager.cc:1045] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.005s
I0619 00:11:57.959954 25068 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 175 ms (attempt = 4)
W0619 00:11:57.960222 25068 catalog_manager.cc:2910] Async tablet task 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
I0619 00:11:57.953085 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:57.964787 25419 leader_election.cc:306] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:57.958735 25741 ts_tablet_manager.cc:1369] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:57.968140 25068 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 4099 ms (attempt = 9)
W0619 00:11:57.968384 25068 catalog_manager.cc:2910] Async tablet task cbd67634f5704165b5bb948501697f0c (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cbd67634f5704165b5bb948501697f0c consensus state has no leader
I0619 00:11:57.968767 26205 raft_consensus.cc:457] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.969069 26205 raft_consensus.cc:2732] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.625s (starting election)
I0619 00:11:57.971333 25084 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC with a delay of 61 ms (attempt = 1)
W0619 00:11:57.971587 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e5e691e6d374cd4bba1f18d8c821723 consensus state has no leader
I0619 00:11:57.974709 25730 ts_tablet_manager.cc:1382] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:57.974964 25730 log.cc:986] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/b34043f14f1c4edd97a1d4689ced5b32
I0619 00:11:57.974789 25727 raft_consensus.cc:628] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:57.975482 26151 raft_consensus.cc:457] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:57.975482 25730 ts_tablet_manager.cc:1403] T b34043f14f1c4edd97a1d4689ced5b32 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:57.976457 25336 ts_tablet_manager.cc:1173] T 507f7ed270af42ce8f404d55b5cae468 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:57.975067 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 53728.9us
I0619 00:11:57.977499 25736 ts_tablet_manager.cc:1382] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I0619 00:11:57.977823 25736 log.cc:986] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/075076aaa16e4ea8a596a7a8c52f103e
I0619 00:11:57.978425 26092 consensus_queue.cc:963] T 19ad911ec6704aef85d9b7f172f08bdc P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:57.978832 25727 consensus_queue.cc:227] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.984200 25084 catalog_manager.cc:3663] Got report from deleted tablet edfdedbd4eca4b62be24b1649f820d0e (table default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:57.984859 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:57.985471 25725 tablet_bootstrap.cc:427] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:57.985827 25725 ts_tablet_manager.cc:1045] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.228s	user 0.005s	sys 0.005s
I0619 00:11:57.987104 25725 raft_consensus.cc:325] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.987695 25725 raft_consensus.cc:351] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:57.987869 25725 raft_consensus.cc:628] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:57.988274 25725 consensus_queue.cc:227] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.989351 25084 catalog_manager.cc:3663] Got report from deleted tablet edfdedbd4eca4b62be24b1649f820d0e (table default.table_1563f330df024855ab3d01187bdc7661 [id=3036c4ccdeae40afbb6be65f6f61f921]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:57.989970 25725 ts_tablet_manager.cc:1073] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.001s
I0619 00:11:57.990716 25725 ts_tablet_manager.cc:1028] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:57.991084 25725 tablet_bootstrap.cc:427] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:57.991019 25068 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 152 ms (attempt = 4)
W0619 00:11:57.991298 25068 catalog_manager.cc:2910] Async tablet task 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 has no consensus state
I0619 00:11:57.993464 25725 tablet_bootstrap.cc:578] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:57.995311 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:57.995895 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:57.998052 25290 leader_election.cc:393] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:57.978440 25736 ts_tablet_manager.cc:1403] T 075076aaa16e4ea8a596a7a8c52f103e P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:57.998311 25290 leader_election.cc:270] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:57.999680 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 299) took 3351ms. Request Metrics: {"tablet-delete.queue_time_us":3112161}
I0619 00:11:58.000407 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 300) took 3346ms. Request Metrics: {"tablet-delete.queue_time_us":3341419}
I0619 00:11:58.001222 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 1035 ms (attempt = 7)
W0619 00:11:58.001498 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:58.001143 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 302) took 3315ms. Request Metrics: {"tablet-delete.queue_time_us":3307228}
I0619 00:11:57.975803 26151 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.027s (starting election)
I0619 00:11:58.002713 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_50568eb659cd47ada4c22ba29a63e552"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:58.002198 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 309) took 3016ms. Request Metrics: {"tablet-delete.queue_time_us":3012408}
W0619 00:11:58.003661 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_50568eb659cd47ada4c22ba29a63e552 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.005201 25082 catalog_manager.cc:3663] Got report from deleted tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c): Sending delete request for this tablet
I0619 00:11:58.005410 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_37643042c7dd406790243851c24ade21" } new_table_name: "default.table_d8b2d7e633094119b0fa40e3972594f8" alter_external_catalogs: true
I0619 00:11:58.005630 25082 catalog_manager.cc:3786] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:58.006608 25082 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]). Expected version 1 got 0
I0619 00:11:58.007019 25082 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]). Expected version 1 got 0
I0619 00:11:58.010062 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_824bd974d1ae4fc398285c3a2393333f" }
I0619 00:11:58.010720 25727 ts_tablet_manager.cc:1073] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.113s	user 0.003s	sys 0.001s
I0619 00:11:58.012426 25069 catalog_manager.cc:2891] Scheduling retry of c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC with a delay of 561 ms (attempt = 6)
W0619 00:11:58.012713 25069 catalog_manager.cc:2910] Async tablet task c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c244dec62a5b4ee492a9a110b656a561 consensus state has no leader
I0619 00:11:58.016160 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) successfully deleted
W0619 00:11:58.016788 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
W0619 00:11:58.017249 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet fc7686daae274e0dba87703fe6924805 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc7686daae274e0dba87703fe6924805 already in progress: creating tablet
I0619 00:11:58.017581 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 50 ms (attempt = 1)
W0619 00:11:58.018044 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb because the tablet was not found. No further retry: Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:58.019109 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.003726 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 310) took 3015ms. Request Metrics: {"tablet-delete.queue_time_us":3011842}
I0619 00:11:58.020296 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 313) took 2974ms. Request Metrics: {"tablet-delete.queue_time_us":2973320}
I0619 00:11:58.020973 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 314) took 2970ms. Request Metrics: {"tablet-delete.queue_time_us":2969884}
I0619 00:11:58.021385 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet cb3c60fa2d8f4cafa12a5b0bc806433f from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:58.021616 25736 tablet_replica.cc:261] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:58.021672 25558 leader_election.cc:379] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.021895 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.021983 25558 leader_election.cc:270] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:58.022109 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.022303 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.022502 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(981d862d1bab4b14bb10e6fc2174e1cc)
W0619 00:11:58.030874 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet b34043f14f1c4edd97a1d4689ced5b32 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b34043f14f1c4edd97a1d4689ced5b32 already in progress: creating tablet
I0619 00:11:58.026394 25462 tablet_service.cc:759] Processing CreateTablet for tablet d54ad9c0bce74297904c77015aa3a6f5 (table=default.table_17ebb06ba3d041478a1d93d0cb9e3964 [id=f210db76e1ce48e58de89fccc6a89b10]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.031112 25071 catalog_manager.cc:2891] Scheduling retry of b34043f14f1c4edd97a1d4689ced5b32 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 27 ms (attempt = 1)
I0619 00:11:58.031519 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d54ad9c0bce74297904c77015aa3a6f5. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:58.031711 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a4c653a255984d3bac88df63362b9334 because the tablet was not found. No further retry: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
I0619 00:11:58.031635 25464 tablet_service.cc:759] Processing CreateTablet for tablet 2af5a8b8f14846aba6c9f104f25e8a21 (table=default.table_ac07448d0aa64bf7b1c84f5011880149 [id=e9b10a248e834d7799ee22e8a507eb31]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.032240 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2af5a8b8f14846aba6c9f104f25e8a21. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:58.032413 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 075076aaa16e4ea8a596a7a8c52f103e because the tablet was not found. No further retry: Not found: Tablet not found: 075076aaa16e4ea8a596a7a8c52f103e
I0619 00:11:58.033713 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet ee4c4769f75141f9b2ef864ab8a8d9e0 (table default.table_4c17e965f551446d965e5fa1499cad9a [id=e2a7900f428948bab0aefd132d8874d4]) successfully deleted
I0619 00:11:58.033951 25078 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_25bd5ae167b74dea974c7eabdd1cd7b0" }
I0619 00:11:58.034243 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC with a delay of 46 ms (attempt = 2)
W0619 00:11:58.034487 25067 catalog_manager.cc:2910] Async tablet task 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9e5e691e6d374cd4bba1f18d8c821723 consensus state has no leader
I0619 00:11:58.035032 25470 tablet_service.cc:759] Processing CreateTablet for tablet 38c512bb89034576a27eb55bbda35985 (table=default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.035593 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 38c512bb89034576a27eb55bbda35985. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.039757 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.040024 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.040263 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.040477 25736 raft_consensus.cc:2042] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:58.040798 25736 raft_consensus.cc:2069] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.042939 25736 ts_tablet_manager.cc:1369] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.045357 25468 tablet_service.cc:759] Processing CreateTablet for tablet a4c3d22a3237408387518e3061470d01 (table=default.table_f18ea472f11c4efa81d1e00b99a5a22a [id=18a0f5bc9de549e6965f46e858ce817a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.046376 25468 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4c3d22a3237408387518e3061470d01. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.047003 26188 raft_consensus.cc:457] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.047283 26188 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.616s (starting election)
I0619 00:11:58.047564 26188 raft_consensus.cc:479] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:11:58.032490 26124 raft_consensus.cc:2732] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.790s (election complete)
I0619 00:11:58.048252 26124 raft_consensus.cc:2784] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.048939 26188 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:11:58.049562 26188 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:11:58.033999 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.052001 25725 tablet_bootstrap.cc:427] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:58.052417 25725 ts_tablet_manager.cc:1045] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.061s	user 0.005s	sys 0.000s
I0619 00:11:58.032245 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.033293 25323 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.053752 25725 raft_consensus.cc:325] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.054251 26124 raft_consensus.cc:2487] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.054342 25725 raft_consensus.cc:351] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.054524 25725 raft_consensus.cc:628] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:58.055022 25725 consensus_queue.cc:227] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:57.961585 25724 raft_consensus.cc:325] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.056623 25724 raft_consensus.cc:351] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.056830 25725 ts_tablet_manager.cc:1073] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:11:58.056845 25724 raft_consensus.cc:628] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:58.057152 25725 ts_tablet_manager.cc:1028] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:58.057471 25725 tablet_bootstrap.cc:427] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:58.059818 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.060948 25470 ts_tablet_manager.cc:1173] T 38c512bb89034576a27eb55bbda35985 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.061390 25725 tablet_bootstrap.cc:578] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:58.002636 26151 raft_consensus.cc:479] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.064134 25727 ts_tablet_manager.cc:1028] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:58.064497 25727 tablet_bootstrap.cc:427] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:58.065367 26151 leader_election.cc:243] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:58.034384 25290 leader_election.cc:306] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:58.064723 25724 consensus_queue.cc:227] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.067371 25323 tablet_service.cc:800] Processing DeleteTablet for tablet edfdedbd4eca4b62be24b1649f820d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.068152 26151 leader_election.cc:243] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.068593 26151 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.074s (election complete)
I0619 00:11:58.069517 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.070444 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cfd335c18c047fba2facc5194807abb" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.064116 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 220) took 3571ms. Request Metrics: {"mutex_wait_us":55977,"tablet-delete.queue_time_us":3263624}
I0619 00:11:58.068827 26151 raft_consensus.cc:2487] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:58.076849 25553 leader_election.cc:306] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.077021 26151 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.144s (election complete)
I0619 00:11:58.077610 26151 raft_consensus.cc:2539] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:58.077929 26151 raft_consensus.cc:593] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Becoming Leader. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Running, Role: LEADER
I0619 00:11:57.969532 26205 raft_consensus.cc:479] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.076910 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 218) took 3588ms. Request Metrics: {"tablet-delete.queue_time_us":3568759}
I0619 00:11:58.080124 25601 tablet_service.cc:759] Processing CreateTablet for tablet d54ad9c0bce74297904c77015aa3a6f5 (table=default.table_17ebb06ba3d041478a1d93d0cb9e3964 [id=f210db76e1ce48e58de89fccc6a89b10]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.080261 25599 tablet_service.cc:759] Processing CreateTablet for tablet 2af5a8b8f14846aba6c9f104f25e8a21 (table=default.table_ac07448d0aa64bf7b1c84f5011880149 [id=e9b10a248e834d7799ee22e8a507eb31]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.081178 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet b34043f14f1c4edd97a1d4689ced5b32 (table default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]) successfully deleted
I0619 00:11:58.082123 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC with a delay of 107 ms (attempt = 3)
W0619 00:11:58.082409 25067 catalog_manager.cc:2910] Async tablet task 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9e5e691e6d374cd4bba1f18d8c821723 consensus state has no leader
W0619 00:11:58.082779 25067 catalog_manager.cc:2910] Async tablet task e5051fc388d74ad39753c43b0ad25402 (table default.table_9751fcfca6454dd99468df3ffb363e82 [id=8977b9567fe14118b3bc189f6581f264]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet e5051fc388d74ad39753c43b0ad25402 consensus state has no leader
I0619 00:11:58.084303 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 2074 ms (attempt = 8)
W0619 00:11:58.084539 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
W0619 00:11:58.065270 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:58.065752 25470 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.068361 25464 ts_tablet_manager.cc:1173] T 2af5a8b8f14846aba6c9f104f25e8a21 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.070413 25467 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.091063 25462 ts_tablet_manager.cc:1173] T d54ad9c0bce74297904c77015aa3a6f5 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.091096 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 2082 ms (attempt = 8)
W0619 00:11:58.091331 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 consensus state has no leader
I0619 00:11:58.091639 25323 tablet_service.cc:759] Processing CreateTablet for tablet d54ad9c0bce74297904c77015aa3a6f5 (table=default.table_17ebb06ba3d041478a1d93d0cb9e3964 [id=f210db76e1ce48e58de89fccc6a89b10]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.092207 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d54ad9c0bce74297904c77015aa3a6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.097035 25468 ts_tablet_manager.cc:1173] T a4c3d22a3237408387518e3061470d01 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.080092 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 223) took 3414ms. Request Metrics: {"tablet-delete.queue_time_us":3412952}
I0619 00:11:58.103099 25351 raft_consensus.cc:2784] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.107203 25741 ts_tablet_manager.cc:1382] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:58.100811 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 225) took 3401ms. Request Metrics: {"tablet-delete.queue_time_us":3400028}
I0619 00:11:58.109292 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d54ad9c0bce74297904c77015aa3a6f5. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:58.113620 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 101720us
I0619 00:11:58.080745 26151 consensus_queue.cc:204] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.111487 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 228) took 3411ms. Request Metrics: {"tablet-delete.queue_time_us":3408305}
I0619 00:11:58.114971 25730 tablet_replica.cc:261] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:58.115728 25598 tablet_service.cc:759] Processing CreateTablet for tablet a4c3d22a3237408387518e3061470d01 (table=default.table_f18ea472f11c4efa81d1e00b99a5a22a [id=18a0f5bc9de549e6965f46e858ce817a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.117974 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
I0619 00:11:58.118690 25082 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 58 ms (attempt = 1)
W0619 00:11:58.118942 25082 catalog_manager.cc:3945] Failed to send 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
W0619 00:11:58.119449 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a284677c4b7f46e18e25edee6024e3eb because the tablet was not found. No further retry: Not found: Tablet not found: a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:58.115200 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.121485 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.121708 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.121896 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.122102 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.122306 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.122501 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.122706 25730 raft_consensus.cc:2042] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:58.123121 25067 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 519 ms (attempt = 6)
W0619 00:11:58.123368 25067 catalog_manager.cc:2910] Async tablet task 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 consensus state has no leader
I0619 00:11:58.110595 25351 raft_consensus.cc:1081] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:58.125916 25947 consensus_queue.cc:963] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:58.128044 25730 raft_consensus.cc:2069] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.112479 25727 tablet_bootstrap.cc:578] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:58.117540 25336 tablet_service.cc:759] Processing CreateTablet for tablet 2af5a8b8f14846aba6c9f104f25e8a21 (table=default.table_ac07448d0aa64bf7b1c84f5011880149 [id=e9b10a248e834d7799ee22e8a507eb31]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:58.119130 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:58.129633 25602 tablet_service.cc:759] Processing CreateTablet for tablet 38c512bb89034576a27eb55bbda35985 (table=default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.120152 25330 tablet_service.cc:759] Processing CreateTablet for tablet 38c512bb89034576a27eb55bbda35985 (table=default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.130293 25736 ts_tablet_manager.cc:1382] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:58.123355 25290 leader_election.cc:379] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.132568 25290 leader_election.cc:270] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:58.132772 25736 log.cc:986] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:58.133450 26124 raft_consensus.cc:2732] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.645s (election complete)
I0619 00:11:58.133697 26124 raft_consensus.cc:2539] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:58.133975 26124 raft_consensus.cc:593] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:58.134392 26124 consensus_queue.cc:204] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.116658 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0070a115e8540cda5f999394deac616" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:58.135844 25622 raft_consensus.cc:2175] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.135879 25736 ts_tablet_manager.cc:1403] T 981d862d1bab4b14bb10e6fc2174e1cc P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:58.137334 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 316) took 3047ms. Request Metrics: {"tablet-delete.queue_time_us":2931008}
I0619 00:11:58.111726 25323 ts_tablet_manager.cc:1173] T d54ad9c0bce74297904c77015aa3a6f5 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.141228 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 38c512bb89034576a27eb55bbda35985. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.130342 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2af5a8b8f14846aba6c9f104f25e8a21. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.144958 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2af5a8b8f14846aba6c9f104f25e8a21. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.152606 25082 master_service.cc:523] ReplaceTablet: received request to replace tablet 0ef73ff0abf44e5aa66cee31180c083c from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:58.153602 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 54373917567a479d93dc8a3fd006d2fe because the tablet was not found. No further retry: Not found: Tablet not found: 54373917567a479d93dc8a3fd006d2fe
W0619 00:11:58.154503 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 2b82f9befb9144a7b4be080ac6d92a73 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 2b82f9befb9144a7b4be080ac6d92a73 already in progress: creating tablet
I0619 00:11:58.157045 25079 catalog_manager.cc:3663] Got report from deleted tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.157364 25079 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.157635 25079 catalog_manager.cc:3663] Got report from deleted tablet 3e7724f967ae41d7b523055d1a4f58db (table default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.157728 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet cb3c60fa2d8f4cafa12a5b0bc806433f
I0619 00:11:58.157945 25079 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:58.158015 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb3c60fa2d8f4cafa12a5b0bc806433f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6)
I0619 00:11:58.158416 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb3c60fa2d8f4cafa12a5b0bc806433f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6)
I0619 00:11:58.120930 26205 leader_election.cc:243] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.158800 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb3c60fa2d8f4cafa12a5b0bc806433f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6)
I0619 00:11:58.159088 25081 catalog_manager.cc:4596] ReplaceTablet: tablet cb3c60fa2d8f4cafa12a5b0bc806433f deleted and replaced by tablet 8252076f40e1409681d92ba4cca0b0e6
I0619 00:11:58.159286 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.159623 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.159956 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.160254 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:58.160354 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 38c512bb89034576a27eb55bbda35985. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.161168 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 0a7470bfeab041f9895e491231b57eb7
I0619 00:11:58.143335 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cfd335c18c047fba2facc5194807abb" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.161485 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.161887 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.162227 25736 tablet_replica.cc:261] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:58.158368 25468 tablet_service.cc:800] Processing DeleteTablet for tablet 075076aaa16e4ea8a596a7a8c52f103e with delete_type TABLET_DATA_DELETED (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.162255 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.163270 25351 raft_consensus.cc:1081] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:58.164757 25085 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 27 ms (attempt = 1)
W0619 00:11:58.165105 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:11:58.164887 25947 consensus_queue.cc:963] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:58.166816 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.167064 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.167294 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.167316 25085 catalog_manager.cc:3663] Got report from deleted tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) (Table deleted at 2018-06-19 00:11:52 PDT): Sending delete request for this tablet
I0619 00:11:58.167570 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.167870 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.167970 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:11:58.168117 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.169127 25068 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 281 ms (attempt = 5)
W0619 00:11:58.169425 25068 catalog_manager.cc:2910] Async tablet task 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
I0619 00:11:58.169803 25068 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 297 ms (attempt = 5)
I0619 00:11:58.170163 25725 tablet_bootstrap.cc:427] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
W0619 00:11:58.170217 25068 catalog_manager.cc:2910] Async tablet task 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 consensus state has no leader
I0619 00:11:58.170295 26206 raft_consensus.cc:457] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.170547 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 2079 ms (attempt = 8)
I0619 00:11:58.170586 26206 raft_consensus.cc:2732] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.900s (starting election)
I0619 00:11:58.170647 25725 ts_tablet_manager.cc:1045] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.113s	user 0.007s	sys 0.007s
W0619 00:11:58.170814 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:58.171654 25741 log.cc:986] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:58.172060 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4c3d22a3237408387518e3061470d01. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.170908 26206 raft_consensus.cc:479] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.174175 26206 leader_election.cc:243] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.174376 25741 ts_tablet_manager.cc:1403] T a52cbcbe35374d518bca1e0fc24d88d4 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:58.175530 26204 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.175817 26204 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.810s (starting election)
I0619 00:11:58.168382 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:58.180047 25736 raft_consensus.cc:2042] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:58.180290 25736 consensus_peers.cc:456] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:58.180532 25736 consensus_peers.cc:456] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:58.180629 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 45 ms (attempt = 2)
I0619 00:11:58.180991 25736 raft_consensus.cc:2069] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.179530 25330 ts_tablet_manager.cc:1173] T 38c512bb89034576a27eb55bbda35985 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:58.181176 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:58.182476 25736 ts_tablet_manager.cc:1369] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.183056 25336 ts_tablet_manager.cc:1173] T 2af5a8b8f14846aba6c9f104f25e8a21 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:58.183419 25558 leader_election.cc:306] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.183719 25558 leader_election.cc:270] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:58.183725 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 60391.9us
I0619 00:11:58.176071 26204 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:58.187115 25058 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.24.119.1:32951, user_credentials={real_user=jenkins}} blocked reactor thread for 87687.1us
I0619 00:11:58.187415 25078 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet ed3b0dace2e2475aa466039465b44a84
I0619 00:11:58.187746 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ed3b0dace2e2475aa466039465b44a84 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.187733 25622 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e5e691e6d374cd4bba1f18d8c821723" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:58.188124 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ed3b0dace2e2475aa466039465b44a84 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.188160 25622 raft_consensus.cc:2784] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.188479 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ed3b0dace2e2475aa466039465b44a84 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.188585 25947 raft_consensus.cc:457] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.188858 25947 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.797s (starting election)
I0619 00:11:58.186029 25618 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:58.188938 25725 raft_consensus.cc:325] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.189497 25618 raft_consensus.cc:2784] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.189779 25725 raft_consensus.cc:351] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.189857 25618 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.932s (vote granted)
I0619 00:11:58.189996 25725 raft_consensus.cc:628] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
W0619 00:11:58.190855 25290 leader_election.cc:306] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:58.190521 25725 consensus_queue.cc:227] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.188383 25622 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.458s (vote granted)
I0619 00:11:58.189106 25947 raft_consensus.cc:479] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.192566 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f155a418d0d547f7b775360f1a90ce01" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.193234 25947 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.193428 25069 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 42 ms (attempt = 2)
W0619 00:11:58.193717 25069 catalog_manager.cc:2910] Async tablet task 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:11:58.194083 25290 leader_election.cc:270] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.194118 25947 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.194622 26124 raft_consensus.cc:2732] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.857s (election complete)
I0619 00:11:58.194850 26124 raft_consensus.cc:2487] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.191352 26206 leader_election.cc:243] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.195742 26206 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.136s (election complete)
I0619 00:11:58.195966 26206 raft_consensus.cc:2487] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:58.193825 25553 leader_election.cc:306] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.185381 26205 leader_election.cc:243] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.196600 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC with a delay of 164 ms (attempt = 4)
W0619 00:11:58.196949 25067 catalog_manager.cc:2910] Async tablet task 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9e5e691e6d374cd4bba1f18d8c821723 consensus state has no leader
I0619 00:11:58.197427 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.154963 25071 catalog_manager.cc:2891] Scheduling retry of 2b82f9befb9144a7b4be080ac6d92a73 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 62 ms (attempt = 1)
I0619 00:11:58.199523 25336 tablet_service.cc:759] Processing CreateTablet for tablet a4c3d22a3237408387518e3061470d01 (table=default.table_f18ea472f11c4efa81d1e00b99a5a22a [id=18a0f5bc9de549e6965f46e858ce817a]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:58.199816 25071 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 45393.5us
I0619 00:11:58.200251 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4c3d22a3237408387518e3061470d01. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.201481 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) successfully deleted
I0619 00:11:58.192373 26209 raft_consensus.cc:457] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:58.202986 25419 leader_election.cc:306] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.177733 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 290) took 4007ms. Request Metrics: {"tablet-delete.queue_time_us":3767929}
I0619 00:11:58.204833 25330 tablet_service.cc:800] Processing DeleteTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f with delete_type TABLET_DATA_DELETED (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.208854 25725 ts_tablet_manager.cc:1073] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.000s
I0619 00:11:58.209383 25602 ts_tablet_manager.cc:1173] T 38c512bb89034576a27eb55bbda35985 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.206585 25078 catalog_manager.cc:3663] Got report from deleted tablet d3c7114381c0423982b9ccbff120c2a2 (table default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:58.214447 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 1068 ms (attempt = 7)
I0619 00:11:58.214427 25622 raft_consensus.cc:2245] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:58.214699 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d3c7114381c0423982b9ccbff120c2a2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
W0619 00:11:58.214596 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 54373917567a479d93dc8a3fd006d2fe because the tablet was not found. No further retry: Not found: Tablet not found: 54373917567a479d93dc8a3fd006d2fe
I0619 00:11:58.202347 26209 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.141s (starting election)
I0619 00:11:58.216364 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_08c6a59c13db436bac657c6610d3ba2d"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.217433 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_08c6a59c13db436bac657c6610d3ba2d in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:58.218997 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:58.205442 25947 raft_consensus.cc:457] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.215673 25468 tablet_service.cc:800] Processing DeleteTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f with delete_type TABLET_DATA_DELETED (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.216389 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.216930 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.217236 25467 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7724f967ae41d7b523055d1a4f58db with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.217537 25470 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.219852 25947 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.103s (starting election)
I0619 00:11:58.218097 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.218690 25465 tablet_service.cc:800] Processing DeleteTablet for tablet d3c7114381c0423982b9ccbff120c2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.219557 25601 ts_tablet_manager.cc:1173] T d54ad9c0bce74297904c77015aa3a6f5 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.218284 25469 tablet_service.cc:800] Processing DeleteTablet for tablet ed3b0dace2e2475aa466039465b44a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.220327 25947 raft_consensus.cc:479] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.222261 25947 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.215674 26209 raft_consensus.cc:479] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.221391 25725 ts_tablet_manager.cc:1028] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:58.223983 25725 tablet_bootstrap.cc:427] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:58.223927 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.224941 25078 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_a06bcbc25e4f4009bb30ac5941a0f8d4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.225558 25419 leader_election.cc:306] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:58.225855 25078 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a06bcbc25e4f4009bb30ac5941a0f8d4 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.226433 25947 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.229249 25598 ts_tablet_manager.cc:1173] T a4c3d22a3237408387518e3061470d01 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.230460 26209 leader_election.cc:243] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.237104 25069 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 81 ms (attempt = 3)
W0619 00:11:58.237367 25069 catalog_manager.cc:2910] Async tablet task 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:11:58.231063 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 108 ms (attempt = 3)
I0619 00:11:58.225136 26204 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.239727 26204 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.240223 26209 leader_election.cc:243] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.240656 25601 tablet_service.cc:800] Processing DeleteTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f with delete_type TABLET_DATA_DELETED (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.241117 25601 tablet_service.cc:800] Processing DeleteTablet for tablet ed3b0dace2e2475aa466039465b44a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:58.241106 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:58.241982 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.245757 25599 ts_tablet_manager.cc:1173] T 2af5a8b8f14846aba6c9f104f25e8a21 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.247905 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.251130 25290 leader_election.cc:393] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.251385 25290 leader_election.cc:270] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.251842 26124 raft_consensus.cc:2732] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.785s (election complete)
I0619 00:11:58.251663 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_7ee2ca99e4784301bfb7f6dbbdc2df9e" } new_table_name: "default.table_54592b380c5f49acac05077eb5114c27" alter_external_catalogs: true
I0619 00:11:58.252056 26124 raft_consensus.cc:2487] T e0070a115e8540cda5f999394deac616 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.253111 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 981d862d1bab4b14bb10e6fc2174e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:52 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.253532 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.254741 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6040e369e0fb445fb993d12dd883039b" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.255847 25730 ts_tablet_manager.cc:1369] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:58.257328 25553 leader_election.cc:306] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.257464 25350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.257828 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "29ee1c11105e4d1e9ac74a247cd2953d" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:58.258954 25553 leader_election.cc:306] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.213325 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 291) took 4042ms. Request Metrics: {"tablet-delete.queue_time_us":4033594}
W0619 00:11:58.259649 25553 leader_election.cc:306] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.259559 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 292) took 4038ms. Request Metrics: {"tablet-delete.queue_time_us":4037933}
I0619 00:11:58.259641 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:58.260222 25741 tablet_replica.cc:261] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:58.260208 25616 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.281s (vote granted)
I0619 00:11:58.260443 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:58.260478 25616 raft_consensus.cc:2245] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:58.260661 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:58.260849 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:58.261034 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:58.261246 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:58.261440 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:58.261644 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(075076aaa16e4ea8a596a7a8c52f103e)
I0619 00:11:58.261814 25741 raft_consensus.cc:2042] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:58.262068 25741 raft_consensus.cc:2069] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.254051 25618 raft_consensus.cc:2245] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:58.264449 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6040e369e0fb445fb993d12dd883039b" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.265120 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f155a418d0d547f7b775360f1a90ce01" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.266418 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "29ee1c11105e4d1e9ac74a247cd2953d" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.267324 25725 tablet_bootstrap.cc:578] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:11:58.269371 25290 leader_election.cc:306] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:58.270650 25330 tablet_service.cc:800] Processing DeleteTablet for tablet ed3b0dace2e2475aa466039465b44a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.271445 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 1059 ms (attempt = 7)
W0619 00:11:58.271785 25071 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:58.272125 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.276113 25079 catalog_manager.cc:3663] Got report from deleted tablet e0070a115e8540cda5f999394deac616 (table default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.276536 25079 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]). Expected version 1 got 0
I0619 00:11:58.277405 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.278203 25724 ts_tablet_manager.cc:1073] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.319s	user 0.004s	sys 0.006s
I0619 00:11:58.278729 25724 ts_tablet_manager.cc:1028] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:58.278946 26124 raft_consensus.cc:457] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.279165 25724 tablet_bootstrap.cc:427] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:58.279170 26124 raft_consensus.cc:2732] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.790s (starting election)
I0619 00:11:58.279415 26124 raft_consensus.cc:479] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.280413 26124 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.280817 26124 leader_election.cc:243] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.281469 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_c5846407e88e4f36b378a06980f17f52"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.282357 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c5846407e88e4f36b378a06980f17f52 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.284057 25724 tablet_bootstrap.cc:578] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:58.286726 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet a52cbcbe35374d518bca1e0fc24d88d4 (table default.table_8687a04df04444c6a3f710f1b9884ce5 [id=c857914eef2b4b74828b67746881df9d]) successfully deleted
I0619 00:11:58.289186 25725 tablet_bootstrap.cc:427] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:58.289520 25725 ts_tablet_manager.cc:1045] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.066s	user 0.004s	sys 0.001s
I0619 00:11:58.293730 25730 ts_tablet_manager.cc:1382] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:58.294046 25730 log.cc:986] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/86959e7dda374ba9ac23d3c1fd30d993
I0619 00:11:58.294699 25730 ts_tablet_manager.cc:1403] T 86959e7dda374ba9ac23d3c1fd30d993 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:58.296389 25736 ts_tablet_manager.cc:1382] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:58.296674 25736 log.cc:986] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/86959e7dda374ba9ac23d3c1fd30d993
I0619 00:11:58.297214 25736 ts_tablet_manager.cc:1403] T 86959e7dda374ba9ac23d3c1fd30d993 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
W0619 00:11:58.297313 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet e60ecc51c92c461889b360dde1dc5cf2 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet e60ecc51c92c461889b360dde1dc5cf2 already in progress: creating tablet
I0619 00:11:58.297539 25067 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 49 ms (attempt = 1)
W0619 00:11:58.297935 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a52cbcbe35374d518bca1e0fc24d88d4 because the tablet was not found. No further retry: Not found: Tablet not found: a52cbcbe35374d518bca1e0fc24d88d4
I0619 00:11:58.299963 25330 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.301386 26226 tablet.cc:1195] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
I0619 00:11:58.304301 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:58.306273 25558 leader_election.cc:306] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.307077 25558 leader_election.cc:270] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.307673 26223 raft_consensus.cc:2732] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.057s (election complete)
I0619 00:11:58.307956 26223 raft_consensus.cc:2487] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.309435 25078 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.310119 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.310360 25724 tablet_bootstrap.cc:427] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:58.310904 25724 ts_tablet_manager.cc:1045] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.032s	user 0.005s	sys 0.000s
I0619 00:11:58.313016 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.314463 25724 raft_consensus.cc:325] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.315469 25724 raft_consensus.cc:351] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.315781 25724 raft_consensus.cc:628] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:58.317127 25727 tablet_bootstrap.cc:427] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:58.317468 25727 ts_tablet_manager.cc:1045] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.253s	user 0.002s	sys 0.008s
I0619 00:11:58.317407 26226 tablet.cc:1195] T 7277051d52dd4be1a6dfa06488c98293 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 2 got alter request for version 2
I0619 00:11:58.318827 25727 raft_consensus.cc:325] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.319413 25727 raft_consensus.cc:351] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.319885 25069 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 136 ms (attempt = 4)
W0619 00:11:58.320155 25069 catalog_manager.cc:2910] Async tablet task 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:11:58.319588 25727 raft_consensus.cc:628] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:58.321130 25727 consensus_queue.cc:227] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.321825 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 317) took 3228ms. Request Metrics: {"tablet-delete.queue_time_us":3068312}
I0619 00:11:58.322479 25736 tablet_replica.cc:261] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:58.322728 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.322930 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.323130 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.323338 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.298831 25725 raft_consensus.cc:325] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.323693 25725 raft_consensus.cc:351] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:58.313741 26220 log.cc:667] Time spent T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I0619 00:11:58.304256 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.323925 25725 raft_consensus.cc:628] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:58.324131 25490 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.805s (vote granted)
I0619 00:11:58.324383 25490 raft_consensus.cc:2245] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
I0619 00:11:58.324637 25727 ts_tablet_manager.cc:1073] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I0619 00:11:58.324476 25725 consensus_queue.cc:227] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.325008 25727 ts_tablet_manager.cc:1028] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:58.325330 25727 tablet_bootstrap.cc:427] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:58.328980 25947 raft_consensus.cc:457] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.306823 25423 leader_election.cc:379] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.329236 25947 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.031s (starting election)
I0619 00:11:58.329447 25423 leader_election.cc:270] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:58.329963 26145 raft_consensus.cc:2732] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.651s (election complete)
I0619 00:11:58.330219 26145 raft_consensus.cc:2539] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:58.330530 26145 raft_consensus.cc:593] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:58.330986 26145 consensus_queue.cc:204] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.329514 25947 raft_consensus.cc:479] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.333825 25947 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.334264 25947 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.336563 26124 raft_consensus.cc:457] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.336834 26124 raft_consensus.cc:2732] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.643s (starting election)
I0619 00:11:58.337074 26124 raft_consensus.cc:479] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.338403 26124 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.338778 26124 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.339340 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 1056 ms (attempt = 7)
W0619 00:11:58.339637 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:11:58.329043 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 224) took 3625ms. Request Metrics: {"mutex_wait_us":5034,"tablet-delete.queue_time_us":3404467}
I0619 00:11:58.340685 25741 ts_tablet_manager.cc:1369] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.341362 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) successfully deleted
I0619 00:11:58.343577 25423 leader_election.cc:379] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.327333 25725 ts_tablet_manager.cc:1073] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.000s
I0619 00:11:58.346411 25423 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:58.346633 25725 ts_tablet_manager.cc:1028] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:58.347050 25725 tablet_bootstrap.cc:427] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:58.316514 25724 consensus_queue.cc:227] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.340801 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 232) took 3527ms. Request Metrics: {"tablet-delete.queue_time_us":3522668}
I0619 00:11:58.348780 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:58.348888 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 231) took 3535ms. Request Metrics: {"tablet-delete.queue_time_us":3508457}
I0619 00:11:58.349561 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 233) took 3502ms. Request Metrics: {"tablet-delete.queue_time_us":3501110}
I0619 00:11:58.350123 25730 tablet_replica.cc:261] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:58.351050 25078 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:58.351692 25079 catalog_manager.cc:3663] Got report from deleted tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d): Sending delete request for this tablet
I0619 00:11:58.351747 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.343586 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_0666108ed61a422aac531a42773d228d [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) successfully deleted
I0619 00:11:58.352342 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
W0619 00:11:58.353094 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc because the tablet was not found. No further retry: Not found: Tablet not found: 981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:58.354898 25727 tablet_bootstrap.cc:578] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
W0619 00:11:58.355886 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc because the tablet was not found. No further retry: Not found: Tablet not found: 981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:58.356830 25469 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:58.356928 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet b8bc8d4a970843258ab5e9530f425d0f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b8bc8d4a970843258ab5e9530f425d0f already in progress: creating tablet
I0619 00:11:58.323570 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.355062 25724 ts_tablet_manager.cc:1073] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.044s	user 0.005s	sys 0.000s
I0619 00:11:58.360108 25724 ts_tablet_manager.cc:1028] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:58.360472 25724 tablet_bootstrap.cc:427] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:58.360666 25071 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 19 ms (attempt = 1)
I0619 00:11:58.359174 26231 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.827s (election complete)
I0619 00:11:58.359719 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.358745 26223 raft_consensus.cc:457] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:58.364465 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:58.364486 26223 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.247s (starting election)
I0619 00:11:58.364758 26223 raft_consensus.cc:479] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:58.365520 25290 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: d62715e230154e2abe126ef97feec7ce
I0619 00:11:58.365798 26223 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.366194 26223 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.367440 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb3c60fa2d8f4cafa12a5b0bc806433f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:58.368965 25553 leader_election.cc:306] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:58.369539 25290 leader_election.cc:306] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: d62715e230154e2abe126ef97feec7ce
I0619 00:11:58.369810 25290 leader_election.cc:270] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.360994 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:58.363819 26231 raft_consensus.cc:2539] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:58.370774 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:58.371060 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:58.371300 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:58.365800 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.371575 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:58.371840 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:58.372063 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(f1b566b2b58f4955aa25ae990a4a8a1a)
I0619 00:11:58.372267 25730 raft_consensus.cc:2042] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:58.372331 25085 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 31 ms (attempt = 1)
W0619 00:11:58.372634 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:11:58.372601 25730 raft_consensus.cc:2069] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.374112 25079 catalog_manager.cc:3663] Got report from deleted tablet f155a418d0d547f7b775360f1a90ce01 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55): Sending delete request for this tablet
I0619 00:11:58.374872 25079 catalog_manager.cc:3786] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 395763cf325d4ce19859830a7ad35c85 (127.24.119.3). New cstate: current_term: 1 leader_uuid: "395763cf325d4ce19859830a7ad35c85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: HEALTHY } } }
I0619 00:11:58.376710 26230 tablet.cc:1200] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:58.353157 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 165 ms (attempt = 4)
W0619 00:11:58.379406 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:58.379781 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 2065 ms (attempt = 8)
W0619 00:11:58.380026 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:11:58.380661 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC with a delay of 270 ms (attempt = 5)
I0619 00:11:58.371007 26231 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W0619 00:11:58.380904 25067 catalog_manager.cc:2910] Async tablet task 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 9e5e691e6d374cd4bba1f18d8c821723 consensus state has no leader
I0619 00:11:58.380903 26231 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.018s (starting election)
I0619 00:11:58.381121 26231 raft_consensus.cc:2784] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.382776 25725 tablet_bootstrap.cc:578] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:58.371529 25736 raft_consensus.cc:2042] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:58.384120 25736 consensus_peers.cc:456] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:58.384343 25736 consensus_peers.cc:456] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:58.384456 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.384876 25736 raft_consensus.cc:2069] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:58.385959 25558 leader_election.cc:306] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.386534 25423 leader_election.cc:379] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.386840 25423 leader_election.cc:270] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:58.388027 26145 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.589s (election complete)
I0619 00:11:58.388283 26145 raft_consensus.cc:2539] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:58.389032 26145 raft_consensus.cc:593] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:58.389472 26145 consensus_queue.cc:204] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.392182 25336 ts_tablet_manager.cc:1173] T a4c3d22a3237408387518e3061470d01 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.386042 25598 tablet_service.cc:800] Processing DeleteTablet for tablet b8bc8d4a970843258ab5e9530f425d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.397023 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f155a418d0d547f7b775360f1a90ce01 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55)
I0619 00:11:58.398957 25598 tablet_service.cc:800] Processing DeleteTablet for tablet f155a418d0d547f7b775360f1a90ce01 with delete_type TABLET_DATA_DELETED (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.400889 25940 raft_consensus.cc:2732] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 3.285s (election complete)
I0619 00:11:58.401121 25940 raft_consensus.cc:2487] T d62715e230154e2abe126ef97feec7ce P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.401131 25558 leader_election.cc:270] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.401695 26223 raft_consensus.cc:2732] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.872s (election complete)
I0619 00:11:58.401893 26223 raft_consensus.cc:2487] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.402971 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.403954 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb3c60fa2d8f4cafa12a5b0bc806433f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.408133 25330 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:58.408735 25290 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
W0619 00:11:58.409227 25290 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
I0619 00:11:58.409163 25724 tablet_bootstrap.cc:578] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:58.409454 25290 leader_election.cc:270] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.410025 25068 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 80 ms (attempt = 2)
W0619 00:11:58.410290 25068 catalog_manager.cc:2910] Async tablet task 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:11:58.413761 25336 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.414639 26226 tablet.cc:1200] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:58.416319 25727 tablet_bootstrap.cc:427] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:58.416568 25079 catalog_manager.cc:3663] Got report from deleted tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_ab760159a5834462a648c3167c0456d9 [id=e064506301e54738aada2d76e8bab056]) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d): Sending delete request for this tablet
I0619 00:11:58.416704 25727 ts_tablet_manager.cc:1045] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.091s	user 0.006s	sys 0.000s
I0619 00:11:58.416894 25079 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.417775 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:11:58.418140 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:11:58.420413 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:58.420960 25730 ts_tablet_manager.cc:1369] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.423197 25940 raft_consensus.cc:2732] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.897s (election complete)
I0619 00:11:58.423439 25940 raft_consensus.cc:2487] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.429224 25336 tablet_service.cc:800] Processing DeleteTablet for tablet d6c3c7f783cb43d59b46502261aeced3 with delete_type TABLET_DATA_DELETED (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.430289 25727 raft_consensus.cc:325] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.430658 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ad726834d8e24054b8d217e0bc82629f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:58.431141 25727 raft_consensus.cc:351] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:58.431552 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ad726834d8e24054b8d217e0bc82629f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.431499 25727 raft_consensus.cc:628] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
W0619 00:11:58.420534 25558 leader_election.cc:306] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.432108 25727 consensus_queue.cc:227] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.435407 25727 ts_tablet_manager.cc:1073] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.000s
W0619 00:11:58.442278 25419 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.1:32951, user_credentials={real_user=jenkins}} blocked reactor thread for 110244us
I0619 00:11:58.444330 25736 ts_tablet_manager.cc:1369] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.446519 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc7686daae274e0dba87703fe6924805" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.449839 26092 raft_consensus.cc:457] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.450071 26092 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.004s (starting election)
I0619 00:11:58.450317 26092 raft_consensus.cc:479] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.451285 26092 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.451696 26092 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.453099 25081 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_2af4c8dc619643729ba189a0dd8cd15f" }
I0619 00:11:58.454357 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_042bb1895e44487bbf4563658402bf87" } new_table_name: "default.table_ed122503385b4360827d3299026dc91e" alter_external_catalogs: true
I0619 00:11:58.454530 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_5f896be5c2ab4c36bbf4ce4bcb10e345" }
I0619 00:11:58.459085 25069 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 286 ms (attempt = 5)
W0619 00:11:58.459368 25069 catalog_manager.cc:2910] Async tablet task 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:11:58.460537 26231 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.461503 26231 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.461905 26231 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.466769 25068 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 542 ms (attempt = 6)
W0619 00:11:58.467056 25068 catalog_manager.cc:2910] Async tablet task 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
W0619 00:11:58.469427 25419 leader_election.cc:306] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.470615 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.475425 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:58.477389 25419 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.477828 25067 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC with a delay of 1065 ms (attempt = 7)
W0619 00:11:58.478121 25067 catalog_manager.cc:2910] Async tablet task 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 consensus state has no leader
I0619 00:11:57.536772 26140 sys_catalog.cc:337] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:11:58.483207 26140 sys_catalog.cc:340] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:11:58.483556 25082 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC with a delay of 22 ms (attempt = 1)
W0619 00:11:58.483851 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
I0619 00:11:58.484462 26140 sys_catalog.cc:424] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:11:58.484944 25068 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 547 ms (attempt = 6)
W0619 00:11:58.485224 25068 catalog_manager.cc:2910] Async tablet task 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 consensus state has no leader
I0619 00:11:58.486737 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.489763 25741 ts_tablet_manager.cc:1382] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I0619 00:11:58.490062 25741 log.cc:986] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/075076aaa16e4ea8a596a7a8c52f103e
I0619 00:11:58.490695 25741 ts_tablet_manager.cc:1403] T 075076aaa16e4ea8a596a7a8c52f103e P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:58.491976 25068 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 102 ms (attempt = 3)
W0619 00:11:58.492255 25068 catalog_manager.cc:2910] Async tablet task 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:11:58.493178 25727 ts_tablet_manager.cc:1028] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:58.493579 25727 tablet_bootstrap.cc:427] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:58.451727 26223 raft_consensus.cc:457] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.456535 26226 tablet.cc:1195] T 19ad911ec6704aef85d9b7f172f08bdc P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
I0619 00:11:58.494359 26223 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.666s (starting election)
I0619 00:11:58.494622 26223 raft_consensus.cc:479] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.495576 26223 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.478219 25724 tablet_bootstrap.cc:427] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:58.495987 26223 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.496444 25727 tablet_bootstrap.cc:578] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:58.496696 25724 ts_tablet_manager.cc:1045] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.136s	user 0.004s	sys 0.001s
I0619 00:11:58.497117 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_2784daa36b93413aa15c32837edf0490"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.498070 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2784daa36b93413aa15c32837edf0490 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.499361 25558 leader_election.cc:270] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.500114 25078 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:58.500766 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.501037 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 1047 ms (attempt = 7)
W0619 00:11:58.501296 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:58.502547 26223 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.957s (election complete)
I0619 00:11:58.502781 26223 raft_consensus.cc:2487] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:58.503250 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 82736.8us
I0619 00:11:58.506283 25087 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:58.502939 25724 raft_consensus.cc:325] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.506986 25724 raft_consensus.cc:351] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.507277 25724 raft_consensus.cc:628] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:58.506772 25087 catalog_manager.cc:3786] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:11:58.507797 25087 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]). Expected version 1 got 0
I0619 00:11:58.507977 25724 consensus_queue.cc:227] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.510593 25079 catalog_manager.cc:3663] Got report from deleted tablet cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6): Sending delete request for this tablet
I0619 00:11:58.511057 25079 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 19ad911ec6704aef85d9b7f172f08bdc (table default.table_5a3f045aae58498688f23e73d119682e [id=9306f61885914c84acf74557c89d2c81]). Expected version 1 got 0
I0619 00:11:58.511864 25621 raft_consensus.cc:1081] T e0070a115e8540cda5f999394deac616 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:58.512217 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb3c60fa2d8f4cafa12a5b0bc806433f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6)
I0619 00:11:58.514639 25558 leader_election.cc:379] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.479074 25725 tablet_bootstrap.cc:427] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:58.492476 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.520848 25725 ts_tablet_manager.cc:1045] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.174s	user 0.006s	sys 0.002s
I0619 00:11:58.522214 25725 raft_consensus.cc:325] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.524229 25725 raft_consensus.cc:351] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.524442 25725 raft_consensus.cc:628] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:58.524930 25725 consensus_queue.cc:227] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.525553 25469 tablet_service.cc:759] Processing CreateTablet for tablet e02218979bd44f6a9f0a950df728b8cd (table=default.table_73944be4ee474c9e9e3f654a4ed45b55 [id=32da28d6d91c42908da1d31fba364d50]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.526244 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e02218979bd44f6a9f0a950df728b8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.526602 25725 ts_tablet_manager.cc:1073] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
W0619 00:11:58.526944 25419 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.1:32951, user_credentials={real_user=jenkins}} blocked reactor thread for 49567.5us
I0619 00:11:58.527499 25725 ts_tablet_manager.cc:1028] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:58.527997 25725 tablet_bootstrap.cc:427] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:58.530390 25419 leader_election.cc:306] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.535049 25469 ts_tablet_manager.cc:1173] T e02218979bd44f6a9f0a950df728b8cd P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.536401 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.541172 25940 raft_consensus.cc:457] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.541390 25940 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.030s (starting election)
I0619 00:11:58.541623 25940 raft_consensus.cc:479] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.542608 25940 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.543009 25940 leader_election.cc:243] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.481127 26235 tablet.cc:1200] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:58.513422 25078 master_service.cc:523] ReplaceTablet: received request to replace tablet bda4e1ed3e8441ba90f63b78f67c61c5 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:58.549538 25465 tablet_service.cc:759] Processing CreateTablet for tablet 4f16f5010ab34867a98ffbb2e5b0a866 (table=default.table_ea3c06e3d9864c2b81bd1aa252b3281a [id=c79f70155b9341f2afff1109cfe53755]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.550195 25465 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4f16f5010ab34867a98ffbb2e5b0a866. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:58.551990 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:58.513880 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_a894c2fb24d74fc99270126b7197d3b1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.553141 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a894c2fb24d74fc99270126b7197d3b1 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.540746 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 1b40a55533f74f5eb90497d3c1064034
I0619 00:11:58.541296 25081 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 1cfd335c18c047fba2facc5194807abb
I0619 00:11:58.553973 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1b40a55533f74f5eb90497d3c1064034 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.553987 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1cfd335c18c047fba2facc5194807abb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.554450 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1cfd335c18c047fba2facc5194807abb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.554476 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1b40a55533f74f5eb90497d3c1064034 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.554875 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1cfd335c18c047fba2facc5194807abb on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.554898 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1b40a55533f74f5eb90497d3c1064034 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.558243 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.558845 25486 raft_consensus.cc:2206] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.531046 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 294) took 4287ms. Request Metrics: {"tablet-delete.queue_time_us":4006211}
I0619 00:11:58.514899 25067 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 1055 ms (attempt = 7)
W0619 00:11:58.563450 25067 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:58.564085 25741 tablet_replica.cc:261] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:58.564149 25067 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC with a delay of 34 ms (attempt = 2)
I0619 00:11:58.564337 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(981d862d1bab4b14bb10e6fc2174e1cc)
W0619 00:11:58.564402 25067 catalog_manager.cc:2910] Async tablet task 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
I0619 00:11:58.564551 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.564784 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.564761 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 298 ms (attempt = 5)
I0619 00:11:58.564991 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(981d862d1bab4b14bb10e6fc2174e1cc)
W0619 00:11:58.565001 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:58.565225 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.565430 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.565650 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(981d862d1bab4b14bb10e6fc2174e1cc)
I0619 00:11:58.568209 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 1cfd335c18c047fba2facc5194807abb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.570129 25465 ts_tablet_manager.cc:1173] T 4f16f5010ab34867a98ffbb2e5b0a866 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:58.570062 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:11:58.575968 25725 tablet_bootstrap.cc:578] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:58.576328 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 1b40a55533f74f5eb90497d3c1064034 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.577231 25084 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_50568eb659cd47ada4c22ba29a63e552" } new_table_name: "default.table_9dfa98569b4d490895c509806ae31ce6" alter_external_catalogs: true
I0619 00:11:58.578969 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.579927 25225 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712" is_pre_election: true
I0619 00:11:58.580508 25225 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.567867 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.571143 25330 tablet_service.cc:759] Processing CreateTablet for tablet e02218979bd44f6a9f0a950df728b8cd (table=default.table_73944be4ee474c9e9e3f654a4ed45b55 [id=32da28d6d91c42908da1d31fba364d50]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.584980 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e02218979bd44f6a9f0a950df728b8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.587497 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 075076aaa16e4ea8a596a7a8c52f103e (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) successfully deleted
I0619 00:11:58.574955 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:58.588783 25748 master_main.cc:86] Master server successfully started.
I0619 00:11:58.586138 25890 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058). Message: Invalid argument: T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.589174 25087 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490] alter complete (version 1)
I0619 00:11:58.589488 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:11:58.575453 25290 consensus_peers.cc:434] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet 981d862d1bab4b14bb10e6fc2174e1cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 981d862d1bab4b14bb10e6fc2174e1cc. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:58.592731 25940 raft_consensus.cc:457] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.592952 25940 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.123s (starting election)
I0619 00:11:58.593179 25940 raft_consensus.cc:479] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.594107 25940 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.594470 25940 leader_election.cc:243] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.594506 25469 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.595393 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 1cfd335c18c047fba2facc5194807abb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.596341 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 1b40a55533f74f5eb90497d3c1064034 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.596843 25290 leader_election.cc:393] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.598382 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d6c3c7f783cb43d59b46502261aeced3" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.598989 25486 raft_consensus.cc:2175] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
W0619 00:11:58.599895 25290 leader_election.cc:306] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: d6c3c7f783cb43d59b46502261aeced3
I0619 00:11:58.602370 25068 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 137 ms (attempt = 4)
W0619 00:11:58.602691 25068 catalog_manager.cc:2910] Async tablet task 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:11:58.521080 25598 tablet_service.cc:759] Processing CreateTablet for tablet e02218979bd44f6a9f0a950df728b8cd (table=default.table_73944be4ee474c9e9e3f654a4ed45b55 [id=32da28d6d91c42908da1d31fba364d50]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:58.537817 25553 leader_election.cc:306] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:58.604019 25553 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.1:32951, user_credentials={real_user=jenkins}} blocked reactor thread for 66219.9us
W0619 00:11:58.604138 25419 consensus_peers.cc:434] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 9e5e691e6d374cd4bba1f18d8c821723. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:58.604269 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e02218979bd44f6a9f0a950df728b8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.554827 25599 tablet_service.cc:800] Processing DeleteTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f with delete_type TABLET_DATA_DELETED (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.557153 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 1b40a55533f74f5eb90497d3c1064034 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.557200 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 1cfd335c18c047fba2facc5194807abb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.604619 26230 tablet.cc:1195] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
I0619 00:11:58.541720 25724 ts_tablet_manager.cc:1073] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.045s	user 0.004s	sys 0.000s
I0619 00:11:58.607502 25724 ts_tablet_manager.cc:1028] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:58.553385 25621 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a7470bfeab041f9895e491231b57eb7" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:58.608086 25621 raft_consensus.cc:2175] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.608853 25067 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC with a delay of 109 ms (attempt = 3)
W0619 00:11:58.609196 25067 catalog_manager.cc:2910] Async tablet task 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
I0619 00:11:58.608094 25724 tablet_bootstrap.cc:427] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:58.611743 25558 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:58.612203 26223 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.147s (election complete)
I0619 00:11:58.612440 26223 raft_consensus.cc:2539] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:58.612665 26223 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:58.612860 26223 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.032s (starting election)
I0619 00:11:58.613054 26223 raft_consensus.cc:2784] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.614841 25598 ts_tablet_manager.cc:1173] T e02218979bd44f6a9f0a950df728b8cd P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.616282 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_cf1ddefe056a429fa21c2b9baf7236a9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.617210 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cf1ddefe056a429fa21c2b9baf7236a9 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.617322 25081 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:58.617853 25081 catalog_manager.cc:3663] Got report from deleted tablet e0070a115e8540cda5f999394deac616 (table default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.617740 26223 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.618118 25081 catalog_manager.cc:3663] Got report from deleted tablet b34043f14f1c4edd97a1d4689ced5b32 (table default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:58.618841 26223 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.619272 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.619833 26223 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.620664 25085 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:58.621114 25085 catalog_manager.cc:3663] Got report from deleted tablet e0070a115e8540cda5f999394deac616 (table default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.621423 25724 tablet_bootstrap.cc:578] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:58.622103 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.622445 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.619621 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.624122 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.624428 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:11:58.625020 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.625174 25730 ts_tablet_manager.cc:1382] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:58.625797 25892 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.625900 25730 log.cc:986] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/f1b566b2b58f4955aa25ae990a4a8a1a
I0619 00:11:58.626107 25892 leader_election.cc:270] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.626471 25730 ts_tablet_manager.cc:1403] T f1b566b2b58f4955aa25ae990a4a8a1a P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:58.627998 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 236) took 3663ms. Request Metrics: {"tablet-delete.queue_time_us":3382841}
I0619 00:11:58.630538 25470 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.631029 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.631665 25465 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:58.635848 25084 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 41 ms (attempt = 1)
W0619 00:11:58.636096 25084 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef has no consensus state
I0619 00:11:58.642081 25079 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.642393 25079 catalog_manager.cc:3663] Got report from deleted tablet 93930ffe285d48d080a8ca4793e871d9 (table default.table_3f22e2fb4a9e47aca16e414746eecd63 [id=acb8add3e0fd49fab202731910e3d20e]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:58.642581 25470 tablet_service.cc:759] Processing CreateTablet for tablet 1b40a55533f74f5eb90497d3c1064034 (table=default.table_5f896be5c2ab4c36bbf4ce4bcb10e345 [id=1dff105fde294540b3f018846cc22c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.643184 25470 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1b40a55533f74f5eb90497d3c1064034. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.643242 25725 tablet_bootstrap.cc:427] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:58.643522 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.643929 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:58.645694 25736 ts_tablet_manager.cc:1382] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:58.645961 25736 log.cc:986] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/54adb588955b45b79de9752f254728ea
I0619 00:11:58.646510 25736 ts_tablet_manager.cc:1403] T 54adb588955b45b79de9752f254728ea P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:58.646857 25725 ts_tablet_manager.cc:1045] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.119s	user 0.007s	sys 0.001s
I0619 00:11:58.647869 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 315) took 3559ms. Request Metrics: {"tablet-delete.queue_time_us":3225168}
I0619 00:11:58.648286 25725 raft_consensus.cc:325] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.614958 25741 raft_consensus.cc:2042] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:58.606998 25330 ts_tablet_manager.cc:1173] T e02218979bd44f6a9f0a950df728b8cd P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.649631 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:58.638852 26235 tablet.cc:1195] T c244dec62a5b4ee492a9a110b656a561 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
I0619 00:11:58.649372 25741 consensus_peers.cc:456] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:58.652493 25741 consensus_peers.cc:456] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
I0619 00:11:58.652931 25741 pending_rounds.cc:63] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Trying to abort 1 pending transactions.
I0619 00:11:58.653100 25741 pending_rounds.cc:68] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264390935084163072 op_type: NO_OP noop_request { }
I0619 00:11:58.653411 25741 raft_consensus.cc:2622] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:11:58.653687 25741 raft_consensus.cc:2069] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.636540 25727 tablet_bootstrap.cc:427] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:58.655395 25727 ts_tablet_manager.cc:1045] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.162s	user 0.005s	sys 0.004s
I0619 00:11:58.656255 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) successfully deleted
I0619 00:11:58.656750 25727 raft_consensus.cc:325] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.660931 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_e5b8888e6fdb4ede8f40117d47639b8b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.661938 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e5b8888e6fdb4ede8f40117d47639b8b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.664942 25067 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 1030 ms (attempt = 7)
W0619 00:11:58.665220 25067 catalog_manager.cc:2910] Async tablet task 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 consensus state has no leader
I0619 00:11:58.633994 25602 tablet_service.cc:759] Processing CreateTablet for tablet 248dd7541f7e4f43ba916a888bab98e1 (table=default.table_4b709074f52242faa541fac2119805ed [id=44cd3f9c9bdc46348e2d00b5c26c1be7]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.668081 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 248dd7541f7e4f43ba916a888bab98e1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.645928 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.646298 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.669206 25727 raft_consensus.cc:351] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.669420 25727 raft_consensus.cc:628] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:58.669976 25727 consensus_queue.cc:227] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.658491 25741 ts_tablet_manager.cc:1369] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.634985 25601 tablet_service.cc:759] Processing CreateTablet for tablet fe93c5dd62d94bdda099b9131482c033 (table=default.table_06ca188d31e6404585c64d19afe3dc8a [id=0de4d1b62e8047a399488a3d3d6831cb]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.671079 25601 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fe93c5dd62d94bdda099b9131482c033. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:58.651028 25553 leader_election.cc:306] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:11:58.612306 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 97688.3us
I0619 00:11:58.673712 25621 raft_consensus.cc:1081] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W0619 00:11:58.649350 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 49465.5us
I0619 00:11:58.674814 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.675397 25330 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.675593 26151 raft_consensus.cc:457] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.675843 26151 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.361s (starting election)
I0619 00:11:58.676252 25290 leader_election.cc:393] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.676074 26151 raft_consensus.cc:479] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.676532 25290 leader_election.cc:270] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.676990 25940 raft_consensus.cc:2732] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.617s (election complete)
I0619 00:11:58.677073 26151 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.677233 25940 raft_consensus.cc:2784] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.677445 26151 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:58.679428 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:11:58.680121 25558 leader_election.cc:306] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.680389 25558 leader_election.cc:270] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.681787 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.682011 26151 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.193s (election complete)
I0619 00:11:58.682225 26151 raft_consensus.cc:2487] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.682459 25940 raft_consensus.cc:2487] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:58.683179 25553 leader_election.cc:306] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.655341 25724 tablet_bootstrap.cc:427] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:58.683966 25724 ts_tablet_manager.cc:1045] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.076s	user 0.005s	sys 0.002s
I0619 00:11:58.684015 25290 leader_election.cc:393] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.649004 25598 tablet_service.cc:759] Processing CreateTablet for tablet 4f16f5010ab34867a98ffbb2e5b0a866 (table=default.table_ea3c06e3d9864c2b81bd1aa252b3281a [id=c79f70155b9341f2afff1109cfe53755]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:58.684695 25071 catalog_manager.cc:2910] Async tablet task f372b9fb174844eca1662c62039a3c71 (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet f372b9fb174844eca1662c62039a3c71 consensus state has no leader
I0619 00:11:58.685353 25724 raft_consensus.cc:325] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.686312 25724 raft_consensus.cc:351] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.686496 25724 raft_consensus.cc:628] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:58.686960 25724 consensus_queue.cc:227] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.688597 25724 ts_tablet_manager.cc:1073] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:11:58.690052 25603 tablet_service.cc:759] Processing CreateTablet for tablet 1b40a55533f74f5eb90497d3c1064034 (table=default.table_5f896be5c2ab4c36bbf4ce4bcb10e345 [id=1dff105fde294540b3f018846cc22c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.694428 25727 ts_tablet_manager.cc:1073] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.000s
I0619 00:11:58.694839 25727 ts_tablet_manager.cc:1028] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:58.695420 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) successfully deleted
I0619 00:11:58.690958 25467 tablet_service.cc:759] Processing CreateTablet for tablet f2d15bd8d6ce460c8422625ba060c6e0 (table=default.table_45234f5ba11946f390d40924c779fe3e [id=12a7e07939ae4125b30d7473b37e1695]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:11:58.696861 25553 consensus_peers.cc:434] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet fc534f75faf345f487abeda223ad88db. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:11:58.697110 25599 tablet_service.cc:759] Processing CreateTablet for tablet f2d15bd8d6ce460c8422625ba060c6e0 (table=default.table_45234f5ba11946f390d40924c779fe3e [id=12a7e07939ae4125b30d7473b37e1695]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.697237 25464 tablet_service.cc:759] Processing CreateTablet for tablet 209b4d3f41df46de9d420e273fa1d1cb (table=default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.697875 25600 tablet_service.cc:759] Processing CreateTablet for tablet 209b4d3f41df46de9d420e273fa1d1cb (table=default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.648545 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 318) took 3551ms. Request Metrics: {"tablet-delete.queue_time_us":3550754}
I0619 00:11:58.697988 26231 consensus_queue.cc:963] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:58.699307 25462 tablet_service.cc:759] Processing CreateTablet for tablet d5c9b2f6c8724888aa4fbe76628d8b7d (table=default.table_3676937cf5b04949a7a870594522a5fa [id=50f16eb3d271463cb301bf0969f0e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.700037 25079 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_4b709074f52242faa541fac2119805ed" } new_table_name: "default.table_034f2a79dabd489ba481a00fccab1d84" alter_external_catalogs: true
I0619 00:11:58.652555 25465 tablet_service.cc:759] Processing CreateTablet for tablet 248dd7541f7e4f43ba916a888bab98e1 (table=default.table_4b709074f52242faa541fac2119805ed [id=44cd3f9c9bdc46348e2d00b5c26c1be7]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.663796 25470 ts_tablet_manager.cc:1173] T 1b40a55533f74f5eb90497d3c1064034 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.701436 25727 tablet_bootstrap.cc:427] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:58.686511 26145 consensus_queue.cc:963] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I0619 00:11:58.702374 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.702386 25487 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.702906 25487 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.650s (vote granted)
I0619 00:11:58.702898 25486 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.834s (vote granted)
I0619 00:11:58.702239 25489 raft_consensus.cc:1081] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer 395763cf325d4ce19859830a7ad35c85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:58.703189 25487 raft_consensus.cc:2245] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
I0619 00:11:58.702149 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 322) took 3485ms. Request Metrics: {"tablet-delete.queue_time_us":3469123}
I0619 00:11:58.703205 25486 raft_consensus.cc:2245] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
I0619 00:11:58.651798 25469 tablet_service.cc:759] Processing CreateTablet for tablet fe93c5dd62d94bdda099b9131482c033 (table=default.table_06ca188d31e6404585c64d19afe3dc8a [id=0de4d1b62e8047a399488a3d3d6831cb]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.704278 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:58.704444 25601 ts_tablet_manager.cc:1173] T fe93c5dd62d94bdda099b9131482c033 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.704845 25490 raft_consensus.cc:2175] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.705335 25725 raft_consensus.cc:351] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.705572 25725 raft_consensus.cc:628] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:58.706260 25725 consensus_queue.cc:227] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.707011 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 209b4d3f41df46de9d420e273fa1d1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.707671 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f2d15bd8d6ce460c8422625ba060c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.709995 25725 ts_tablet_manager.cc:1073] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.063s	user 0.004s	sys 0.000s
I0619 00:11:58.710512 25725 ts_tablet_manager.cc:1028] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:58.710930 25725 tablet_bootstrap.cc:427] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:58.711316 25465 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 248dd7541f7e4f43ba916a888bab98e1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.712337 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 323) took 3482ms. Request Metrics: {"tablet-delete.queue_time_us":3473582}
I0619 00:11:58.712957 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d5c9b2f6c8724888aa4fbe76628d8b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.717774 25724 ts_tablet_manager.cc:1028] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:58.718456 25724 tablet_bootstrap.cc:427] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:58.701607 25067 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 45 ms (attempt = 2)
W0619 00:11:58.722290 25067 catalog_manager.cc:2910] Async tablet task 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef has no consensus state
I0619 00:11:58.718046 25602 ts_tablet_manager.cc:1173] T 248dd7541f7e4f43ba916a888bab98e1 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.723510 25067 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC with a delay of 149 ms (attempt = 4)
I0619 00:11:58.723935 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fe93c5dd62d94bdda099b9131482c033. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:58.723954 25067 catalog_manager.cc:2910] Async tablet task 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
I0619 00:11:58.684258 25290 leader_election.cc:270] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.726469 25940 raft_consensus.cc:2732] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.137s (election complete)
I0619 00:11:58.726749 25940 raft_consensus.cc:2784] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.726812 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 326) took 3440ms. Request Metrics: {"tablet-delete.queue_time_us":3413271}
I0619 00:11:58.727564 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 330) took 3436ms. Request Metrics: {"tablet-delete.queue_time_us":3390956}
I0619 00:11:58.728260 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 331) took 3391ms. Request Metrics: {"tablet-delete.queue_time_us":3389891}
I0619 00:11:58.728899 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 334) took 3363ms. Request Metrics: {"tablet-delete.queue_time_us":3362570}
I0619 00:11:58.729519 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 335) took 3359ms. Request Metrics: {"tablet-delete.queue_time_us":3359073}
I0619 00:11:58.731281 25940 raft_consensus.cc:2487] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.733028 25069 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC with a delay of 4098 ms (attempt = 9)
W0619 00:11:58.733307 25069 catalog_manager.cc:2910] Async tablet task cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet cb3c60fa2d8f4cafa12a5b0bc806433f consensus state has no leader
I0619 00:11:58.732681 25727 tablet_bootstrap.cc:578] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:58.734675 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet 992f255e14544af3b8a89cd295944f29 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:58.735842 25558 leader_election.cc:379] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.736120 25558 leader_election.cc:270] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:58.736569 26223 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.109s (election complete)
I0619 00:11:58.736832 26223 raft_consensus.cc:2539] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:58.737043 26223 raft_consensus.cc:457] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:58.737247 26223 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.042s (starting election)
I0619 00:11:58.737452 26223 raft_consensus.cc:2784] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.738145 25079 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 46 ms (attempt = 1)
W0619 00:11:58.738435 25079 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
I0619 00:11:58.739280 25741 ts_tablet_manager.cc:1382] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:58.742012 25068 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 291 ms (attempt = 5)
I0619 00:11:58.723198 26151 raft_consensus.cc:457] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:11:58.742353 25068 catalog_manager.cc:2910] Async tablet task 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:11:58.745826 25069 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 553 ms (attempt = 6)
W0619 00:11:58.746104 25069 catalog_manager.cc:2910] Async tablet task 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:11:58.750149 25741 log.cc:986] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:58.729899 25464 ts_tablet_manager.cc:1173] T 209b4d3f41df46de9d420e273fa1d1cb P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.741556 25725 tablet_bootstrap.cc:578] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:58.753079 25462 ts_tablet_manager.cc:1173] T d5c9b2f6c8724888aa4fbe76628d8b7d P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:11:58.740169 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 56168.5us
W0619 00:11:58.734702 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 2b82f9befb9144a7b4be080ac6d92a73 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 2b82f9befb9144a7b4be080ac6d92a73 already in progress: creating tablet
I0619 00:11:58.724143 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 235) took 3762ms. Request Metrics: {"tablet-delete.queue_time_us":3722812}
I0619 00:11:58.754467 25071 catalog_manager.cc:2891] Scheduling retry of 2b82f9befb9144a7b4be080ac6d92a73 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 55 ms (attempt = 1)
I0619 00:11:58.754616 26151 raft_consensus.cc:2732] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.927s (starting election)
I0619 00:11:58.754899 26151 raft_consensus.cc:479] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:58.755712 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet d0624684b6b04ba3be260b2705dc00e1 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet d0624684b6b04ba3be260b2705dc00e1 already in progress: creating tablet
I0619 00:11:58.755926 25071 catalog_manager.cc:2891] Scheduling retry of d0624684b6b04ba3be260b2705dc00e1 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 40 ms (attempt = 1)
I0619 00:11:58.744700 26223 raft_consensus.cc:479] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:58.756502 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:58.756809 26223 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:58.756991 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 86959e7dda374ba9ac23d3c1fd30d993 because the tablet was not found. No further retry: Not found: Tablet not found: 86959e7dda374ba9ac23d3c1fd30d993
I0619 00:11:58.757185 26223 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:58.757411 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet b8bc8d4a970843258ab5e9530f425d0f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b8bc8d4a970843258ab5e9530f425d0f already in progress: creating tablet
I0619 00:11:58.757616 25071 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 60 ms (attempt = 1)
W0619 00:11:58.758007 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet d0624684b6b04ba3be260b2705dc00e1 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet d0624684b6b04ba3be260b2705dc00e1 already in progress: creating tablet
I0619 00:11:58.758208 25071 catalog_manager.cc:2891] Scheduling retry of d0624684b6b04ba3be260b2705dc00e1 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 22 ms (attempt = 1)
W0619 00:11:58.758812 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet f1b566b2b58f4955aa25ae990a4a8a1a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f1b566b2b58f4955aa25ae990a4a8a1a already in progress: creating tablet
I0619 00:11:58.758962 25084 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_3c040b90a5ff40c180e4679dee37fd1f" }
I0619 00:11:58.759574 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:58.759601 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_8dda14f89d4a4854a63884924f69321e"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:58.760484 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8dda14f89d4a4854a63884924f69321e in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.761116 26238 catalog_manager.cc:1062] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: acquiring CA information for follower catalog manager: success
I0619 00:11:58.761029 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 336) took 3381ms. Request Metrics: {"tablet-delete.queue_time_us":3380104}
I0619 00:11:58.761579 25736 tablet_replica.cc:261] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:58.761839 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.762055 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.762253 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.762455 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.762709 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.762915 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.763106 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.763279 25736 raft_consensus.cc:2042] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:58.763487 25736 consensus_peers.cc:456] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:58.736941 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
W0619 00:11:58.764904 25553 leader_election.cc:306] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.764875 26151 leader_election.cc:243] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.766314 25602 tablet_service.cc:759] Processing CreateTablet for tablet d5c9b2f6c8724888aa4fbe76628d8b7d (table=default.table_3676937cf5b04949a7a870594522a5fa [id=50f16eb3d271463cb301bf0969f0e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.767133 26151 leader_election.cc:243] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.764250 26223 consensus_queue.cc:963] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I0619 00:11:58.740844 25621 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:58.768664 25330 tablet_service.cc:759] Processing CreateTablet for tablet 4f16f5010ab34867a98ffbb2e5b0a866 (table=default.table_ea3c06e3d9864c2b81bd1aa252b3281a [id=c79f70155b9341f2afff1109cfe53755]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.769235 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4f16f5010ab34867a98ffbb2e5b0a866. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.771114 25067 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 102 ms (attempt = 3)
I0619 00:11:58.765421 25618 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:58.772949 26145 raft_consensus.cc:457] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.773232 26145 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.218s (starting election)
W0619 00:11:58.771859 25067 catalog_manager.cc:2910] Async tablet task 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef has no consensus state
I0619 00:11:58.771975 25621 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.321s (vote granted)
I0619 00:11:58.773718 25618 raft_consensus.cc:2175] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10bda0b4856f4441b5aed24cdc4764a8 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:58.773581 26145 raft_consensus.cc:479] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.774910 26145 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.774729 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "93930ffe285d48d080a8ca4793e871d9" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.775900 26145 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:58.775920 25553 leader_election.cc:306] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.776346 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b34043f14f1c4edd97a1d4689ced5b32" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.777029 25467 ts_tablet_manager.cc:1173] T f2d15bd8d6ce460c8422625ba060c6e0 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.774708 25621 raft_consensus.cc:2245] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
W0619 00:11:58.777580 25419 leader_election.cc:306] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.781265 25730 tablet_replica.cc:261] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:58.781503 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.781733 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.781919 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.782101 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.782320 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.782529 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.782538 25330 ts_tablet_manager.cc:1173] T 4f16f5010ab34867a98ffbb2e5b0a866 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.782579 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 209b4d3f41df46de9d420e273fa1d1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.782752 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:58.782937 25730 raft_consensus.cc:2042] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:58.783196 25730 raft_consensus.cc:2069] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.783957 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4f16f5010ab34867a98ffbb2e5b0a866. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.784371 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f2d15bd8d6ce460c8422625ba060c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.784698 25069 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 43 ms (attempt = 2)
W0619 00:11:58.784965 25069 catalog_manager.cc:2910] Async tablet task 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
I0619 00:11:58.785128 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d5c9b2f6c8724888aa4fbe76628d8b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.786016 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1b40a55533f74f5eb90497d3c1064034. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.786923 25725 tablet_bootstrap.cc:427] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:58.787250 25725 ts_tablet_manager.cc:1045] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.076s	user 0.004s	sys 0.000s
I0619 00:11:58.787341 25336 tablet_service.cc:759] Processing CreateTablet for tablet fe93c5dd62d94bdda099b9131482c033 (table=default.table_06ca188d31e6404585c64d19afe3dc8a [id=0de4d1b62e8047a399488a3d3d6831cb]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.787897 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fe93c5dd62d94bdda099b9131482c033. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.788637 25725 raft_consensus.cc:325] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.791191 25725 raft_consensus.cc:351] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.791894 25725 raft_consensus.cc:628] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:58.797147 25336 ts_tablet_manager.cc:1173] T fe93c5dd62d94bdda099b9131482c033 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.800458 25730 ts_tablet_manager.cc:1369] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.795125 25725 consensus_queue.cc:227] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.803638 26145 raft_consensus.cc:457] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.803649 25323 tablet_service.cc:759] Processing CreateTablet for tablet f2d15bd8d6ce460c8422625ba060c6e0 (table=default.table_45234f5ba11946f390d40924c779fe3e [id=12a7e07939ae4125b30d7473b37e1695]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.803954 26145 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.504s (starting election)
I0619 00:11:58.804208 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f2d15bd8d6ce460c8422625ba060c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.804342 26145 raft_consensus.cc:479] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.806372 26145 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.807021 26145 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.808187 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f1b566b2b58f4955aa25ae990a4a8a1a" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:58.809545 25419 leader_election.cc:306] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.811298 25331 tablet_service.cc:759] Processing CreateTablet for tablet 209b4d3f41df46de9d420e273fa1d1cb (table=default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.811892 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 209b4d3f41df46de9d420e273fa1d1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.795719 25071 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 36 ms (attempt = 1)
W0619 00:11:58.812366 25071 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 53516.1us
I0619 00:11:58.813493 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:11:58.814251 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 2086 ms (attempt = 8)
W0619 00:11:58.814671 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:58.815145 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:11:58.815806 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.816839 25082 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_5e0e10c382ce4a0b8aeb14cc019630a0" }
I0619 00:11:58.816273 25469 ts_tablet_manager.cc:1173] T fe93c5dd62d94bdda099b9131482c033 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.818393 25736 consensus_peers.cc:456] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:58.819198 25736 raft_consensus.cc:2069] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:58.820571 25419 consensus_peers.cc:434] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 9e5e691e6d374cd4bba1f18d8c821723. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:11:58.821002 25331 ts_tablet_manager.cc:1173] T 209b4d3f41df46de9d420e273fa1d1cb P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.823468 25323 ts_tablet_manager.cc:1173] T f2d15bd8d6ce460c8422625ba060c6e0 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.824748 25727 tablet_bootstrap.cc:427] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:58.825103 25727 ts_tablet_manager.cc:1045] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.124s	user 0.005s	sys 0.000s
I0619 00:11:58.827507 26231 raft_consensus.cc:457] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.826448 25727 raft_consensus.cc:325] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.827761 26231 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.905s (starting election)
I0619 00:11:58.827816 25727 raft_consensus.cc:351] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.827987 25727 raft_consensus.cc:628] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:58.828011 26231 raft_consensus.cc:479] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.828531 25727 consensus_queue.cc:227] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.828992 26231 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.829378 26231 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.829453 25069 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 105 ms (attempt = 3)
W0619 00:11:58.829759 25069 catalog_manager.cc:2910] Async tablet task 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
I0619 00:11:58.830143 25068 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 4098 ms (attempt = 9)
W0619 00:11:58.830411 25068 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:58.830976 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.750804 25741 ts_tablet_manager.cc:1403] T 981d862d1bab4b14bb10e6fc2174e1cc P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:58.832973 25419 leader_election.cc:306] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.833035 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 296) took 4575ms. Request Metrics: {"tablet-delete.queue_time_us":4302037}
I0619 00:11:58.834069 25741 tablet_replica.cc:261] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:58.834379 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.834585 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.834830 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.835052 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.835294 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.835520 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.835757 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(54adb588955b45b79de9752f254728ea)
I0619 00:11:58.835944 25741 raft_consensus.cc:2042] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:58.836212 25741 raft_consensus.cc:2069] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.840977 25741 ts_tablet_manager.cc:1369] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.841356 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 981d862d1bab4b14bb10e6fc2174e1cc (table default.table_48d04a9a6f224dd983111f0215fb417b [id=01235cbfa38b48a9bbfbb0b4630c5d23]) successfully deleted
I0619 00:11:58.842748 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 992f255e14544af3b8a89cd295944f29
I0619 00:11:58.843046 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 992f255e14544af3b8a89cd295944f29 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet e5b3be4f05474541b222b0c42884ec80)
I0619 00:11:58.843287 25598 ts_tablet_manager.cc:1173] T 4f16f5010ab34867a98ffbb2e5b0a866 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.843420 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 992f255e14544af3b8a89cd295944f29 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet e5b3be4f05474541b222b0c42884ec80)
I0619 00:11:58.843829 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 992f255e14544af3b8a89cd295944f29 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet e5b3be4f05474541b222b0c42884ec80)
I0619 00:11:58.844132 25087 catalog_manager.cc:4596] ReplaceTablet: tablet 992f255e14544af3b8a89cd295944f29 deleted and replaced by tablet e5b3be4f05474541b222b0c42884ec80
I0619 00:11:58.847944 26238 catalog_manager.cc:1090] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0619 00:11:58.849115 25465 ts_tablet_manager.cc:1173] T 248dd7541f7e4f43ba916a888bab98e1 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.850309 25725 ts_tablet_manager.cc:1073] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.063s	user 0.005s	sys 0.000s
I0619 00:11:58.850710 25725 ts_tablet_manager.cc:1028] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:58.850404 25600 ts_tablet_manager.cc:1173] T 209b4d3f41df46de9d420e273fa1d1cb P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.851052 25725 tablet_bootstrap.cc:427] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:58.852902 25603 ts_tablet_manager.cc:1173] T 1b40a55533f74f5eb90497d3c1064034 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.853636 25725 tablet_bootstrap.cc:578] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:58.853989 25736 ts_tablet_manager.cc:1369] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:58.817436 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7c57c5ca55f24621b66197a3698504bd because the tablet was not found. No further retry: Not found: Tablet not found: 7c57c5ca55f24621b66197a3698504bd
I0619 00:11:58.855628 25558 leader_election.cc:379] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.855917 25558 leader_election.cc:270] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:58.859050 26151 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.641s (election complete)
I0619 00:11:58.859282 26151 raft_consensus.cc:2539] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:58.859457 25558 leader_election.cc:393] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.859760 25558 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:11:58.859632 26151 raft_consensus.cc:457] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:58.860060 26151 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.811s (starting election)
I0619 00:11:58.860038 25423 leader_election.cc:393] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10bda0b4856f4441b5aed24cdc4764a8 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:58.860260 26151 raft_consensus.cc:2784] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.860327 25423 leader_election.cc:270] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.861263 26145 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.670s (election complete)
I0619 00:11:58.861490 26145 raft_consensus.cc:2784] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.861600 25599 ts_tablet_manager.cc:1173] T f2d15bd8d6ce460c8422625ba060c6e0 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.859372 26223 raft_consensus.cc:457] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.864830 26223 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.750s (starting election)
I0619 00:11:58.865347 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 541 ms (attempt = 6)
I0619 00:11:58.865089 26223 raft_consensus.cc:479] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:58.865639 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:58.866029 26223 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.866382 26223 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.861594 25724 tablet_bootstrap.cc:578] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:58.867904 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:58.842224 25336 tablet_service.cc:759] Processing CreateTablet for tablet 1b40a55533f74f5eb90497d3c1064034 (table=default.table_5f896be5c2ab4c36bbf4ce4bcb10e345 [id=1dff105fde294540b3f018846cc22c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.846246 25351 raft_consensus.cc:1081] T 0a7470bfeab041f9895e491231b57eb7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:58.868698 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:58.869565 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1b40a55533f74f5eb90497d3c1064034. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.870497 25423 leader_election.cc:379] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.870810 25423 leader_election.cc:270] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:11:58.871109 25553 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:58.870286 26227 consensus_queue.cc:963] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:58.871610 26231 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.869s (election complete)
I0619 00:11:58.871866 26231 raft_consensus.cc:2539] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:58.872071 26231 raft_consensus.cc:457] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:58.872272 26231 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.135s (starting election)
I0619 00:11:58.872493 26231 raft_consensus.cc:2784] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.848824 25330 tablet_service.cc:759] Processing CreateTablet for tablet 248dd7541f7e4f43ba916a888bab98e1 (table=default.table_4b709074f52242faa541fac2119805ed [id=44cd3f9c9bdc46348e2d00b5c26c1be7]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.873029 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 248dd7541f7e4f43ba916a888bab98e1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.875361 25067 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC with a delay of 304 ms (attempt = 5)
W0619 00:11:58.875638 25067 catalog_manager.cc:2910] Async tablet task 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
I0619 00:11:58.877363 26231 raft_consensus.cc:479] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.878321 26231 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.878770 26231 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.850536 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 992f255e14544af3b8a89cd295944f29 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b3be4f05474541b222b0c42884ec80) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.851099 25323 tablet_service.cc:759] Processing CreateTablet for tablet d5c9b2f6c8724888aa4fbe76628d8b7d (table=default.table_3676937cf5b04949a7a870594522a5fa [id=50f16eb3d271463cb301bf0969f0e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:58.879822 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:58.880259 25351 raft_consensus.cc:2784] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.880295 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d5c9b2f6c8724888aa4fbe76628d8b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:58.880525 25351 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.654s (vote granted)
I0619 00:11:58.881268 26145 raft_consensus.cc:2487] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.881678 25490 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 3.125s (vote granted)
I0619 00:11:58.882330 25067 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 146 ms (attempt = 4)
W0619 00:11:58.882565 25067 catalog_manager.cc:2910] Async tablet task 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef has no consensus state
I0619 00:11:58.883564 25602 ts_tablet_manager.cc:1173] T d5c9b2f6c8724888aa4fbe76628d8b7d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.885929 25490 raft_consensus.cc:2245] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 1.
I0619 00:11:58.885318 25351 raft_consensus.cc:2245] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
W0619 00:11:58.888025 25423 leader_election.cc:306] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: fc7686daae274e0dba87703fe6924805
I0619 00:11:58.888267 25419 leader_election.cc:379] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.888550 25419 leader_election.cc:270] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:58.888617 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "93930ffe285d48d080a8ca4793e871d9" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.889385 26231 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.971s (election complete)
I0619 00:11:58.889489 25730 ts_tablet_manager.cc:1382] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:58.889701 26231 raft_consensus.cc:2539] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:58.889761 25730 log.cc:986] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/0a33940e5347489993ab4679e9aeafde
I0619 00:11:58.831140 25727 ts_tablet_manager.cc:1073] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I0619 00:11:58.890995 26231 raft_consensus.cc:593] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:11:58.886432 26151 raft_consensus.cc:479] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.891453 26231 consensus_queue.cc:204] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.892473 25727 ts_tablet_manager.cc:1028] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:58.893420 25730 ts_tablet_manager.cc:1403] T 0a33940e5347489993ab4679e9aeafde P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:58.894948 25727 tablet_bootstrap.cc:427] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:58.895548 26223 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.816s (election complete)
I0619 00:11:58.894819 26151 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:58.895783 26223 raft_consensus.cc:2487] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.897909 25727 tablet_bootstrap.cc:578] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:58.899533 25085 catalog_manager.cc:3663] Got report from deleted tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:58.900537 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:58.900810 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:58.901273 25623 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.073s (vote granted)
I0619 00:11:58.901489 25623 raft_consensus.cc:2245] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:58.906337 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:58.906968 25351 raft_consensus.cc:2175] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.907984 25553 leader_election.cc:393] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote denied by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951). Message: Invalid argument: T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:58.909533 25078 catalog_manager.cc:3663] Got report from deleted tablet e9a221a1d9814ef99a0b4ccaa04d1017 (table default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1): Sending delete request for this tablet
I0619 00:11:58.910238 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:11:58.911140 25081 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]). Expected version 1 got 0
I0619 00:11:58.911303 25724 tablet_bootstrap.cc:427] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:58.911635 25724 ts_tablet_manager.cc:1045] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.193s	user 0.006s	sys 0.004s
I0619 00:11:58.911869 25081 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 32 ms (attempt = 1)
W0619 00:11:58.912127 25081 catalog_manager.cc:3945] Failed to send 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:58.912792 25940 raft_consensus.cc:457] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:58.913031 25940 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.628s (starting election)
I0619 00:11:58.913262 25940 raft_consensus.cc:479] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.914264 25940 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.914669 25940 leader_election.cc:243] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:58.917218 25084 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 9e5e691e6d374cd4bba1f18d8c821723
I0619 00:11:58.917505 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.917872 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.917901 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:58.918215 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.918437 25623 raft_consensus.cc:2206] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.919343 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.919773 25326 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.919831 25290 leader_election.cc:393] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:58.920858 25724 raft_consensus.cc:325] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.921489 25724 raft_consensus.cc:351] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.923687 26151 leader_election.cc:243] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.925474 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc534f75faf345f487abeda223ad88db" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:58.927057 25336 ts_tablet_manager.cc:1173] T 1b40a55533f74f5eb90497d3c1064034 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.930239 25082 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 48eb7be170a64eb1bd869eba7067843d
I0619 00:11:58.930544 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 48eb7be170a64eb1bd869eba7067843d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.930938 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 48eb7be170a64eb1bd869eba7067843d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.931283 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 48eb7be170a64eb1bd869eba7067843d on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.931715 25724 raft_consensus.cc:628] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:58.932286 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.932729 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:58.935621 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_65c17e22939e4de49d493e144f6bf26f"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:58.936100 25069 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 170 ms (attempt = 4)
W0619 00:11:58.936372 25069 catalog_manager.cc:2910] Async tablet task 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
W0619 00:11:58.936542 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_65c17e22939e4de49d493e144f6bf26f in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.932185 25724 consensus_queue.cc:227] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.939982 25079 catalog_manager.cc:3663] Got report from deleted tablet 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) (Table deleted at 2018-06-19 00:11:57 PDT): Sending delete request for this tablet
I0619 00:11:58.940671 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4feb648bb30247b49a4abd49da95ea7d on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:58.943543 25724 ts_tablet_manager.cc:1073] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.001s
I0619 00:11:58.944114 25490 raft_consensus.cc:2175] T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:58.943862 25724 ts_tablet_manager.cc:1028] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:58.944380 25741 ts_tablet_manager.cc:1382] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:58.945364 25724 tablet_bootstrap.cc:427] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:58.946049 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 32 ms (attempt = 2)
I0619 00:11:58.894846 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 240) took 3770ms. Request Metrics: {"tablet-delete.queue_time_us":3626800}
W0619 00:11:58.946306 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:58.946894 25602 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.946738 25730 tablet_replica.cc:261] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:58.947793 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 992f255e14544af3b8a89cd295944f29 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b3be4f05474541b222b0c42884ec80) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.948060 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:11:58.948287 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:11:58.948499 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:11:58.948755 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:11:58.948899 25602 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.949532 25290 leader_election.cc:393] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T fc534f75faf345f487abeda223ad88db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:58.949853 25290 leader_election.cc:270] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.950234 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.950505 25940 raft_consensus.cc:2732] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.697s (election complete)
I0619 00:11:58.950755 25940 raft_consensus.cc:2784] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:58.951687 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 48eb7be170a64eb1bd869eba7067843d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:58.931501 25330 ts_tablet_manager.cc:1173] T 248dd7541f7e4f43ba916a888bab98e1 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.954457 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_0666108ed61a422aac531a42773d228d" } new_table_name: "default.table_f7cff01960d64d739ae50955fabe7254" alter_external_catalogs: true
I0619 00:11:58.954746 25940 raft_consensus.cc:2487] T fc534f75faf345f487abeda223ad88db P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.888324 25423 leader_election.cc:270] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:58.955850 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:58.956400 25623 raft_consensus.cc:2175] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10bda0b4856f4441b5aed24cdc4764a8 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:58.956902 25323 ts_tablet_manager.cc:1173] T d5c9b2f6c8724888aa4fbe76628d8b7d P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:58.957969 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 48eb7be170a64eb1bd869eba7067843d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.958148 26145 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.593s (election complete)
I0619 00:11:58.958411 26145 raft_consensus.cc:2487] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:58.960566 25623 raft_consensus.cc:1081] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:11:58.961777 25079 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:58.962770 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.964083 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.965245 25741 log.cc:986] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/54adb588955b45b79de9752f254728ea
I0619 00:11:58.965927 25741 ts_tablet_manager.cc:1403] T 54adb588955b45b79de9752f254728ea P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:58.967427 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 297) took 4707ms. Request Metrics: {"tablet-delete.queue_time_us":4571494}
I0619 00:11:58.968134 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 54adb588955b45b79de9752f254728ea (table default.table_531af4913e434ddc8afce2aa7b081d1c [id=b32370a87f5a45b58631d73fead1aa42]) successfully deleted
I0619 00:11:58.968809 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 298) took 4697ms. Request Metrics: {"tablet-delete.queue_time_us":4697250}
I0619 00:11:58.969202 25082 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 58 ms (attempt = 1)
W0619 00:11:58.969486 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
W0619 00:11:58.969981 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a because the tablet was not found. No further retry: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:58.970639 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 299) took 4666ms. Request Metrics: {"tablet-delete.queue_time_us":4665823}
W0619 00:11:58.971333 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 because the tablet was not found. No further retry: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:58.972360 25727 tablet_bootstrap.cc:427] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:58.972793 25727 ts_tablet_manager.cc:1045] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.078s	user 0.005s	sys 0.000s
W0619 00:11:58.972801 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 84788.2us
I0619 00:11:58.974532 25727 raft_consensus.cc:325] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.975149 25727 raft_consensus.cc:351] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:58.975349 25727 raft_consensus.cc:628] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
W0619 00:11:58.975694 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet d62715e230154e2abe126ef97feec7ce because the tablet was not found. No further retry: Not found: Tablet not found: d62715e230154e2abe126ef97feec7ce
I0619 00:11:58.976234 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_37bb1c669e584d62b5b5a9f562e3f4d6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:58.949003 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:11:58.977084 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:11:58.977303 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(6040e369e0fb445fb993d12dd883039b)
W0619 00:11:58.977474 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_37bb1c669e584d62b5b5a9f562e3f4d6 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:58.977479 25730 raft_consensus.cc:2042] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:58.977753 25730 raft_consensus.cc:2069] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:58.979058 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 73 ms (attempt = 3)
W0619 00:11:58.979349 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:58.981478 25730 ts_tablet_manager.cc:1369] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:58.979430 25558 leader_election.cc:379] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:58.985399 25558 leader_election.cc:270] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:58.989553 25079 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:58.990198 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:58.991811 26151 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.370s (election complete)
I0619 00:11:58.992215 26151 raft_consensus.cc:2539] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:58.992259 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:58.992538 26151 raft_consensus.cc:593] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Becoming Leader. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Running, Role: LEADER
I0619 00:11:58.992964 26151 consensus_queue.cc:204] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:58.993891 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 4feb648bb30247b49a4abd49da95ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.002627 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_a6a97f8910184441b129e04f28661796"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.003505 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a6a97f8910184441b129e04f28661796 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.007768 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 300) took 4640ms. Request Metrics: {"tablet-delete.queue_time_us":4637404}
I0619 00:11:59.009452 25724 tablet_bootstrap.cc:578] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:59.006966 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 775f0b6b309c49278066a5e7cf623af2 because the tablet was not found. No further retry: Not found: Tablet not found: 775f0b6b309c49278066a5e7cf623af2
I0619 00:11:59.010710 25068 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC with a delay of 1062 ms (attempt = 7)
I0619 00:11:59.010848 25084 master_service.cc:523] ReplaceTablet: received request to replace tablet edfdedbd4eca4b62be24b1649f820d0e from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:59.010982 25068 catalog_manager.cc:2910] Async tablet task 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
W0619 00:11:59.011703 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet fc7686daae274e0dba87703fe6924805 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet fc7686daae274e0dba87703fe6924805 already in progress: creating tablet
I0619 00:11:59.011931 25067 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 27 ms (attempt = 1)
I0619 00:11:59.008404 25727 consensus_queue.cc:227] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.015703 25727 ts_tablet_manager.cc:1073] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.043s	user 0.004s	sys 0.000s
I0619 00:11:59.019405 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 4131 ms (attempt = 9)
W0619 00:11:59.019693 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.020956 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.021414 25486 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.073s (vote granted)
I0619 00:11:59.021625 25486 raft_consensus.cc:2245] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
I0619 00:11:59.022845 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:59.023103 25465 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.023358 25616 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.074s (vote granted)
W0619 00:11:59.023485 25558 leader_election.cc:306] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.023576 25616 raft_consensus.cc:2245] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:59.023782 25558 leader_election.cc:270] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.024485 25465 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.025756 25727 ts_tablet_manager.cc:1028] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:59.026021 25736 ts_tablet_manager.cc:1382] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:59.012346 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) successfully deleted
I0619 00:11:59.028440 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 78 ms (attempt = 2)
W0619 00:11:59.028729 25067 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:59.029099 25067 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 280 ms (attempt = 5)
W0619 00:11:59.029325 25067 catalog_manager.cc:2910] Async tablet task 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef has no consensus state
I0619 00:11:59.029494 25079 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_ab760159a5834462a648c3167c0456d9" } new_table_name: "default.table_a2078fb395364185b90d58ba4ed9318d" alter_external_catalogs: true
I0619 00:11:59.032490 25727 tablet_bootstrap.cc:427] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:59.012792 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 301) took 4627ms. Request Metrics: {"tablet-delete.queue_time_us":4627231}
W0619 00:11:59.033164 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet cb57d196f0724247b6a7f9a34be8377a because the tablet was not found. No further retry: Not found: Tablet not found: cb57d196f0724247b6a7f9a34be8377a
I0619 00:11:59.033990 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 307) took 4408ms. Request Metrics: {"tablet-delete.queue_time_us":4406145}
I0619 00:11:59.034220 25068 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 1030 ms (attempt = 7)
W0619 00:11:59.034481 25068 catalog_manager.cc:2910] Async tablet task 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 consensus state has no leader
I0619 00:11:59.026463 25736 log.cc:986] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/0a33940e5347489993ab4679e9aeafde
W0619 00:11:59.034756 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet b34043f14f1c4edd97a1d4689ced5b32 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b34043f14f1c4edd97a1d4689ced5b32 already in progress: creating tablet
I0619 00:11:59.034847 25068 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 560 ms (attempt = 6)
I0619 00:11:59.034991 25067 catalog_manager.cc:2891] Scheduling retry of b34043f14f1c4edd97a1d4689ced5b32 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 22 ms (attempt = 1)
W0619 00:11:59.035085 25068 catalog_manager.cc:2910] Async tablet task 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:11:59.037964 25736 ts_tablet_manager.cc:1403] T 0a33940e5347489993ab4679e9aeafde P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:59.038939 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 309) took 4399ms. Request Metrics: {"tablet-delete.queue_time_us":4398577}
I0619 00:11:59.039672 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:11:59.040091 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7c57c5ca55f24621b66197a3698504bd because the tablet was not found. No further retry: Not found: Tablet not found: 7c57c5ca55f24621b66197a3698504bd
I0619 00:11:59.040895 25079 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 29 ms (attempt = 1)
I0619 00:11:59.041085 25323 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:11:59.041249 25079 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.042131 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 2095 ms (attempt = 8)
W0619 00:11:59.042567 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:59.043702 26151 raft_consensus.cc:2732] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.758s (election complete)
I0619 00:11:59.044059 26151 raft_consensus.cc:2487] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.044293 25727 tablet_bootstrap.cc:578] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:59.046385 25741 tablet_replica.cc:261] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:59.046639 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:59.046844 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:59.047722 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 338) took 3665ms. Request Metrics: {"tablet-delete.queue_time_us":3379137}
I0619 00:11:59.048563 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 339) took 3665ms. Request Metrics: {"tablet-delete.queue_time_us":3665081}
I0619 00:11:59.049551 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 340) took 3666ms. Request Metrics: {"tablet-delete.queue_time_us":3665504}
I0619 00:11:59.047025 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:59.050537 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 341) took 3632ms. Request Metrics: {"tablet-delete.queue_time_us":3631421}
I0619 00:11:59.050945 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:59.051689 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 342) took 3618ms. Request Metrics: {"tablet-delete.queue_time_us":3617068}
I0619 00:11:59.053326 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 132 ms (attempt = 4)
I0619 00:11:59.053242 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
W0619 00:11:59.053572 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.051911 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:59.054091 25736 tablet_replica.cc:261] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:59.054267 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:59.054924 25490 raft_consensus.cc:2206] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:59.055860 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(86959e7dda374ba9ac23d3c1fd30d993)
I0619 00:11:59.056406 25741 raft_consensus.cc:2042] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.056690 25741 raft_consensus.cc:2069] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.057477 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:59.050503 25725 tablet_bootstrap.cc:427] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:59.058589 25323 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.059016 25725 ts_tablet_manager.cc:1045] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.208s	user 0.004s	sys 0.001s
I0619 00:11:59.060353 25725 raft_consensus.cc:325] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.061038 25725 raft_consensus.cc:351] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.061246 25725 raft_consensus.cc:628] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:59.061735 25725 consensus_queue.cc:227] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.062916 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:59.063521 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:59.067412 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) successfully deleted
W0619 00:11:59.067988 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 because the tablet was not found. No further retry: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:59.069808 25730 ts_tablet_manager.cc:1382] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:59.070029 25730 log.cc:986] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/6040e369e0fb445fb993d12dd883039b
I0619 00:11:59.070480 25730 ts_tablet_manager.cc:1403] T 6040e369e0fb445fb993d12dd883039b P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:59.064899 25465 tablet_service.cc:800] Processing DeleteTablet for tablet 992f255e14544af3b8a89cd295944f29 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b3be4f05474541b222b0c42884ec80) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.067025 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.072193 26151 raft_consensus.cc:457] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.072237 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 44 ms (attempt = 2)
I0619 00:11:59.072415 26151 raft_consensus.cc:2732] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.684s (starting election)
W0619 00:11:59.072494 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.072607 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:59.072649 26151 raft_consensus.cc:479] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.073447 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:59.073616 26151 leader_election.cc:243] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.074043 26151 leader_election.cc:243] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.075212 25558 leader_election.cc:379] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.075235 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4feb648bb30247b49a4abd49da95ea7d" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:59.075716 25558 leader_election.cc:270] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:59.076625 25724 tablet_bootstrap.cc:427] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
W0619 00:11:59.076700 25553 leader_election.cc:306] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.077013 25724 ts_tablet_manager.cc:1045] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.132s	user 0.004s	sys 0.000s
I0619 00:11:59.077389 26151 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.825s (election complete)
W0619 00:11:59.077610 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 because the tablet was not found. No further retry: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:59.077656 26151 raft_consensus.cc:2539] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:59.077883 26151 raft_consensus.cc:457] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:59.077813 25741 ts_tablet_manager.cc:1369] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.078078 26151 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.806s (starting election)
I0619 00:11:59.078269 26151 raft_consensus.cc:2784] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:59.078552 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 397ae10bb6d841b4ba561ec431d29864 because the tablet was not found. No further retry: Not found: Tablet not found: 397ae10bb6d841b4ba561ec431d29864
I0619 00:11:59.080443 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:59.080823 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:59.081521 25558 leader_election.cc:393] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:59.081816 25558 leader_election.cc:270] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:11:59.081800 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:11:59.081501 25618 raft_consensus.cc:1081] T 0a7470bfeab041f9895e491231b57eb7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:59.082696 25725 ts_tablet_manager.cc:1073] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.023s	user 0.003s	sys 0.001s
I0619 00:11:59.082099 25736 raft_consensus.cc:2042] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.083366 25736 raft_consensus.cc:2069] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:11:59.083477 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 0ef73ff0abf44e5aa66cee31180c083c with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0ef73ff0abf44e5aa66cee31180c083c already in progress: creating tablet
I0619 00:11:59.083765 25071 catalog_manager.cc:2891] Scheduling retry of 0ef73ff0abf44e5aa66cee31180c083c Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 60 ms (attempt = 1)
I0619 00:11:59.084352 25423 leader_election.cc:379] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.084780 25423 leader_election.cc:270] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:59.085250 26231 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.910s (election complete)
I0619 00:11:59.085479 26231 raft_consensus.cc:2539] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:59.085718 26231 raft_consensus.cc:457] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:59.085904 26231 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.186s (starting election)
I0619 00:11:59.086098 26231 raft_consensus.cc:2784] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.093006 25727 tablet_bootstrap.cc:427] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:59.093374 25727 ts_tablet_manager.cc:1045] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.061s	user 0.005s	sys 0.000s
I0619 00:11:59.094456 25736 ts_tablet_manager.cc:1369] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.094799 25727 raft_consensus.cc:325] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.095408 25727 raft_consensus.cc:351] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.095595 25727 raft_consensus.cc:628] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:59.096053 25727 consensus_queue.cc:227] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.098544 25727 ts_tablet_manager.cc:1073] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I0619 00:11:59.098915 25727 ts_tablet_manager.cc:1028] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:59.094559 26231 raft_consensus.cc:479] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.100518 26231 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.102352 26231 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.103693 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 48eb7be170a64eb1bd869eba7067843d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.104578 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4feb648bb30247b49a4abd49da95ea7d" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.105563 25469 tablet_service.cc:800] Processing DeleteTablet for tablet b8bc8d4a970843258ab5e9530f425d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.105486 26151 raft_consensus.cc:479] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.106144 25423 leader_election.cc:393] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10bda0b4856f4441b5aed24cdc4764a8 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:59.106449 25423 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:11:59.106928 26151 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.107079 26231 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.932s (election complete)
I0619 00:11:59.107360 26231 raft_consensus.cc:2487] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.107338 25069 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 266 ms (attempt = 5)
I0619 00:11:59.101950 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 241) took 3946ms. Request Metrics: {"tablet-delete.queue_time_us":3790054}
W0619 00:11:59.107642 25069 catalog_manager.cc:2910] Async tablet task 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
I0619 00:11:59.108065 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 66 ms (attempt = 3)
W0619 00:11:59.108320 25067 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:59.108359 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 6040e369e0fb445fb993d12dd883039b (table default.table_673676f1e1e94d76afe2c9b06d701801 [id=980403da36b04fad89427ba22b96d098]) successfully deleted
I0619 00:11:59.107364 26151 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.110457 26151 raft_consensus.cc:2732] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.713s (election complete)
I0619 00:11:59.110707 26151 raft_consensus.cc:2487] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.083099 25725 ts_tablet_manager.cc:1028] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:59.111819 25725 tablet_bootstrap.cc:427] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:59.114549 25725 tablet_bootstrap.cc:578] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:59.116686 25469 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.117357 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 71 ms (attempt = 3)
W0619 00:11:59.117609 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.119439 26145 consensus_queue.cc:963] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:59.119874 25724 raft_consensus.cc:325] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.120462 25724 raft_consensus.cc:351] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.120666 25724 raft_consensus.cc:628] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:59.101989 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:59.106495 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9d44681acacf4f04b6fcfbd4c9a18e87" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:59.110966 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:59.121089 25724 consensus_queue.cc:227] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:59.123787 25419 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.124125 25727 tablet_bootstrap.cc:427] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:59.108476 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 242) took 3854ms. Request Metrics: {"tablet-delete.queue_time_us":3853629}
W0619 00:11:59.125074 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 0ef73ff0abf44e5aa66cee31180c083c with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0ef73ff0abf44e5aa66cee31180c083c already in progress: creating tablet
I0619 00:11:59.125293 25071 catalog_manager.cc:2891] Scheduling retry of 0ef73ff0abf44e5aa66cee31180c083c Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 48 ms (attempt = 1)
I0619 00:11:59.126798 25727 tablet_bootstrap.cc:578] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
W0619 00:11:59.127579 25423 leader_election.cc:306] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: b34043f14f1c4edd97a1d4689ced5b32
I0619 00:11:59.127868 25423 leader_election.cc:270] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:59.128262 25553 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.128358 26231 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.571s (election complete)
I0619 00:11:59.128669 26231 raft_consensus.cc:2487] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.128726 25724 ts_tablet_manager.cc:1073] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.052s	user 0.005s	sys 0.000s
I0619 00:11:59.129078 25724 ts_tablet_manager.cc:1028] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:59.129416 25724 tablet_bootstrap.cc:427] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:59.130527 25616 raft_consensus.cc:2175] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10bda0b4856f4441b5aed24cdc4764a8 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:59.136909 25724 tablet_bootstrap.cc:578] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:59.136888 25741 ts_tablet_manager.cc:1382] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:59.137120 25741 log.cc:986] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/86959e7dda374ba9ac23d3c1fd30d993
I0619 00:11:59.125952 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 246) took 3847ms. Request Metrics: {"tablet-delete.queue_time_us":3846883}
W0619 00:11:59.139662 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet d6c3c7f783cb43d59b46502261aeced3 because the tablet was not found. No further retry: Not found: Tablet not found: d6c3c7f783cb43d59b46502261aeced3
I0619 00:11:59.140293 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 250) took 3727ms. Request Metrics: {"tablet-delete.queue_time_us":3726543}
I0619 00:11:59.140671 25940 raft_consensus.cc:457] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.140920 25940 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.655s (starting election)
W0619 00:11:59.141012 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet edfdedbd4eca4b62be24b1649f820d0e because the tablet was not found. No further retry: Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
I0619 00:11:59.141166 25940 raft_consensus.cc:479] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.141633 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 249) took 3736ms. Request Metrics: {"tablet-delete.queue_time_us":3727470}
I0619 00:11:59.142177 25940 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:11:59.142297 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet edfdedbd4eca4b62be24b1649f820d0e because the tablet was not found. No further retry: Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
I0619 00:11:59.142591 25940 leader_election.cc:243] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.142706 26151 raft_consensus.cc:457] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.142940 26151 raft_consensus.cc:2732] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.986s (starting election)
I0619 00:11:59.144980 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0ef73ff0abf44e5aa66cee31180c083c on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:59.143175 26151 raft_consensus.cc:479] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:59.146010 25290 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:59.146178 26151 leader_election.cc:243] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:11:59.146477 25290 leader_election.cc:306] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: a284677c4b7f46e18e25edee6024e3eb
I0619 00:11:59.146572 26151 leader_election.cc:243] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.146728 25290 leader_election.cc:270] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.147743 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9a221a1d9814ef99a0b4ccaa04d1017" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:59.148233 26262 tablet.cc:1200] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
W0619 00:11:59.149195 25553 leader_election.cc:306] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.153836 25079 catalog_manager.cc:3663] Got report from deleted tablet 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) (Table deleted at 2018-06-19 00:11:57 PDT): Sending delete request for this tablet
I0619 00:11:59.154240 25079 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet c244dec62a5b4ee492a9a110b656a561 (table default.table_c4520a4e6b374312a9f1801b938645c0 [id=9f88d7e327314448a85e60f6aa682490]). Expected version 1 got 0
I0619 00:11:59.155104 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4feb648bb30247b49a4abd49da95ea7d on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:11:59.161185 25940 raft_consensus.cc:2732] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.878s (election complete)
I0619 00:11:59.161419 25940 raft_consensus.cc:2487] T a284677c4b7f46e18e25edee6024e3eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.162144 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 4feb648bb30247b49a4abd49da95ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:57 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.164263 25730 tablet_replica.cc:261] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:59.164604 25079 catalog_manager.cc:3663] Got report from deleted tablet e0070a115e8540cda5f999394deac616 (table default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:59.164908 25079 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.165197 25079 catalog_manager.cc:3663] Got report from deleted tablet f75b8a4f1fd845fbab6f83f59b7cfc58 (table default.table_fa53e172a2a34136a5a6efb8b5c39368 [id=779e902dec1a4701a176d545a35e9978]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.164691 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.165763 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.165653 25081 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_65c17e22939e4de49d493e144f6bf26f" }
I0619 00:11:59.166189 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.166532 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.166954 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.167291 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.167697 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.169540 25078 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_811056346d9b4f5e8fb98ccc54d55045" }
I0619 00:11:59.174041 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0ef73ff0abf44e5aa66cee31180c083c on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:59.174823 25741 ts_tablet_manager.cc:1403] T 86959e7dda374ba9ac23d3c1fd30d993 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:11:59.175103 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.175429 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 168 ms (attempt = 4)
W0619 00:11:59.175709 25067 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:59.168015 25730 raft_consensus.cc:2042] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.176198 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 310) took 4466ms. Request Metrics: {"tablet-delete.queue_time_us":4333238}
I0619 00:11:59.178251 25081 catalog_manager.cc:4020] Not sending DeleteTablet requests; no consensus state for tablet 804321b0641e4356a418d2c02031b0af
I0619 00:11:59.179312 25730 raft_consensus.cc:2069] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.180224 25067 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC with a delay of 513 ms (attempt = 6)
W0619 00:11:59.180474 25067 catalog_manager.cc:2910] Async tablet task 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
I0619 00:11:59.182994 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 4132 ms (attempt = 9)
W0619 00:11:59.183256 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_33f663222f084af580067f0f88d85725 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:59.185668 25078 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet fb8e94875a8347a0a473c95a77c38795
I0619 00:11:59.181821 25727 tablet_bootstrap.cc:427] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:59.185945 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fb8e94875a8347a0a473c95a77c38795 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.186266 25727 ts_tablet_manager.cc:1045] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.062s	user 0.004s	sys 0.000s
I0619 00:11:59.186316 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fb8e94875a8347a0a473c95a77c38795 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.186707 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fb8e94875a8347a0a473c95a77c38795 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.186686 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 277 ms (attempt = 5)
W0619 00:11:59.187027 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.188079 25323 tablet_service.cc:800] Processing DeleteTablet for tablet fb8e94875a8347a0a473c95a77c38795 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.188385 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:59.188869 25602 tablet_service.cc:800] Processing DeleteTablet for tablet fb8e94875a8347a0a473c95a77c38795 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.189054 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 177 ms (attempt = 4)
I0619 00:11:59.188805 25727 raft_consensus.cc:325] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:59.189311 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.189385 25727 raft_consensus.cc:351] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.189563 25727 raft_consensus.cc:628] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:59.189735 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:59.189944 25602 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.190004 25727 consensus_queue.cc:227] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.190479 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:59.190835 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.191656 25602 tablet_service.cc:800] Processing DeleteTablet for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.192932 25084 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]). Expected version 1 got 0
I0619 00:11:59.193075 25727 ts_tablet_manager.cc:1073] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I0619 00:11:59.193527 25727 ts_tablet_manager.cc:1028] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:59.193635 25084 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 20 ms (attempt = 1)
W0619 00:11:59.193917 25084 catalog_manager.cc:3945] Failed to send 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:59.194072 25727 tablet_bootstrap.cc:427] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:59.196527 25082 catalog_manager.cc:3663] Got report from deleted tablet f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:59.196660 25724 tablet_bootstrap.cc:427] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:59.196976 25724 ts_tablet_manager.cc:1045] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.068s	user 0.005s	sys 0.001s
I0619 00:11:59.196959 25082 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]). Expected version 1 got 0
I0619 00:11:59.197438 25082 catalog_manager.cc:3663] Got report from deleted tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:59.198035 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 86959e7dda374ba9ac23d3c1fd30d993 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) successfully deleted
I0619 00:11:59.198176 25724 raft_consensus.cc:325] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.198637 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:59.198755 25724 raft_consensus.cc:351] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.199059 25082 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 55 ms (attempt = 1)
I0619 00:11:59.198921 25724 raft_consensus.cc:628] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:11:59.199316 25082 catalog_manager.cc:3945] Failed to send 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.199604 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:59.201535 25084 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.201853 25084 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.202641 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.203021 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:59.199934 25724 consensus_queue.cc:227] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.203807 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 311) took 4489ms. Request Metrics: {"tablet-delete.queue_time_us":4476851}
I0619 00:11:59.204427 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 316) took 4458ms. Request Metrics: {"tablet-delete.queue_time_us":4429439}
I0619 00:11:59.206037 25084 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.206573 25084 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]). Expected version 2 got 0
I0619 00:11:59.205348 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 318) took 4379ms. Request Metrics: {"tablet-delete.queue_time_us":4378339}
I0619 00:11:59.207742 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.208222 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.208451 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.209334 25078 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_d9245875d6214b88a63997a3d300f512" } new_table_name: "default.table_c7eb124f548947d7940b1de0352abd23" alter_external_catalogs: true
I0619 00:11:59.209843 25084 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 55 ms (attempt = 1)
I0619 00:11:59.209081 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_5714a0615d4d4f588547fdcf3951168b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.210899 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5714a0615d4d4f588547fdcf3951168b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:59.212049 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 2b82f9befb9144a7b4be080ac6d92a73 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 2b82f9befb9144a7b4be080ac6d92a73 already in progress: creating tablet
I0619 00:11:59.212278 25067 catalog_manager.cc:2891] Scheduling retry of 2b82f9befb9144a7b4be080ac6d92a73 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 50 ms (attempt = 1)
W0619 00:11:59.212672 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7c57c5ca55f24621b66197a3698504bd because the tablet was not found. No further retry: Not found: Tablet not found: 7c57c5ca55f24621b66197a3698504bd
W0619 00:11:59.213100 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 because the tablet was not found. No further retry: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
I0619 00:11:59.214978 25724 ts_tablet_manager.cc:1073] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.000s
I0619 00:11:59.216761 25724 ts_tablet_manager.cc:1028] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
W0619 00:11:59.216863 25084 catalog_manager.cc:3945] Failed to send 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:59.217806 25079 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b7a90cb93e8f4fe79cb29ff3ab818928"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:59.218590 25724 tablet_bootstrap.cc:427] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:11:59.218713 25079 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b7a90cb93e8f4fe79cb29ff3ab818928 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:59.219935 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 92366.4us
I0619 00:11:59.220618 25068 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 32 ms (attempt = 2)
W0619 00:11:59.220909 25068 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:59.207559 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 317) took 4381ms. Request Metrics: {"tablet-delete.queue_time_us":4369467}
I0619 00:11:59.222991 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 319) took 4374ms. Request Metrics: {"tablet-delete.queue_time_us":4372498}
W0619 00:11:59.223743 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 03522c48c50943b38cae5aa88562e583 because the tablet was not found. No further retry: Not found: Tablet not found: 03522c48c50943b38cae5aa88562e583
I0619 00:11:59.224213 25081 catalog_manager.cc:3663] Got report from deleted tablet 8e3a356933e749b9a4ade476ec30f475 (table default.table_2d9df18dc333494d9398874c17e04acf [id=0d90b232db8e48ed9bfab0c59c6fbdba]) (replaced by tablet beddf1523a854f5b9292c0e4013d7436): Sending delete request for this tablet
I0619 00:11:59.224880 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
I0619 00:11:59.225598 25725 tablet_bootstrap.cc:427] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:59.226284 25725 ts_tablet_manager.cc:1045] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.115s	user 0.005s	sys 0.000s
I0619 00:11:59.226438 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 8e3a356933e749b9a4ade476ec30f475 with delete_type TABLET_DATA_DELETED (replaced by tablet beddf1523a854f5b9292c0e4013d7436) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.229076 25725 raft_consensus.cc:325] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.229725 25725 raft_consensus.cc:351] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.229929 25725 raft_consensus.cc:628] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:59.230068 25730 ts_tablet_manager.cc:1369] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.230393 25725 consensus_queue.cc:227] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:59.233255 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 151741us
W0619 00:11:59.235430 25558 leader_election.cc:306] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.235257 25081 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d2bfd9daf88c4770a6ce099bcdfb3931"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:59.235728 25558 leader_election.cc:270] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:11:59.236259 25081 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d2bfd9daf88c4770a6ce099bcdfb3931 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.238181 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 321) took 4319ms. Request Metrics: {"tablet-delete.queue_time_us":4318701}
I0619 00:11:59.239965 25085 master_service.cc:523] ReplaceTablet: received request to replace tablet 3e7724f967ae41d7b523055d1a4f58db from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:59.240660 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet b8bc8d4a970843258ab5e9530f425d0f with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b8bc8d4a970843258ab5e9530f425d0f already in progress: creating tablet
I0619 00:11:59.239565 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ef4ccfddaf014d73b60e77f40fd7944b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:59.241262 25067 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 38 ms (attempt = 1)
W0619 00:11:59.241801 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ef4ccfddaf014d73b60e77f40fd7944b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.242461 25741 tablet_replica.cc:261] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:59.242775 25085 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:11:59.243165 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:59.243423 25724 tablet_bootstrap.cc:578] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:59.243446 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:59.243957 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(0474be659e5e4a4cbfc181b8c812b1eb)
W0619 00:11:59.244470 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7c57c5ca55f24621b66197a3698504bd because the tablet was not found. No further retry: Not found: Tablet not found: 7c57c5ca55f24621b66197a3698504bd
I0619 00:11:59.244971 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_2d9df18dc333494d9398874c17e04acf" } new_table_name: "default.table_b75186c4eb524b428a93fa8d4af3f24e" alter_external_catalogs: true
I0619 00:11:59.245460 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:59.245911 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:59.239444 25727 tablet_bootstrap.cc:578] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:59.237401 26262 tablet.cc:1195] T c244dec62a5b4ee492a9a110b656a561 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
I0619 00:11:59.247860 25940 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.248117 25940 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.555s (starting election)
I0619 00:11:59.248385 25940 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.249097 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:59.251626 26266 raft_consensus.cc:2732] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.230s (election complete)
I0619 00:11:59.251912 26266 raft_consensus.cc:2487] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.254484 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.255573 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 75 ms (attempt = 2)
W0619 00:11:59.255856 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.256357 25087 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:11:59.257078 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:59.258074 25068 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 100 ms (attempt = 3)
I0619 00:11:59.259168 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:59.259754 25490 raft_consensus.cc:2175] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:59.260593 25469 tablet_service.cc:800] Processing DeleteTablet for tablet fb8e94875a8347a0a473c95a77c38795 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.261509 25469 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:59.261945 25068 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
W0619 00:11:59.262341 25423 leader_election.cc:306] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: f1b566b2b58f4955aa25ae990a4a8a1a
I0619 00:11:59.262679 25423 leader_election.cc:270] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.265921 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:59.269112 25602 tablet_service.cc:759] Processing CreateTablet for tablet 9a38dedf214144739519839883b4ca59 (table=default.table_e5b8888e6fdb4ede8f40117d47639b8b [id=6324147fd0974be596a8c79042245c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.269760 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9a38dedf214144739519839883b4ca59. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.272583 25069 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 50 ms (attempt = 2)
W0619 00:11:59.272881 25069 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:59.249358 25940 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.273098 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:59.273314 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(0474be659e5e4a4cbfc181b8c812b1eb)
I0619 00:11:59.273485 25741 raft_consensus.cc:2042] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.273780 25741 raft_consensus.cc:2069] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.273402 25940 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.274920 25741 ts_tablet_manager.cc:1369] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.276087 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:59.276710 25616 raft_consensus.cc:2175] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:59.277855 25290 leader_election.cc:393] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:59.281051 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:59.281682 25598 tablet_service.cc:759] Processing CreateTablet for tablet 536870e2d8d94dbba25bdd0df31bef17 (table=default.table_2784daa36b93413aa15c32837edf0490 [id=b5ff0ee44b064715b2fc34fcbb7ad5b5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.282472 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 536870e2d8d94dbba25bdd0df31bef17. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.288208 25078 catalog_manager.cc:2891] Scheduling retry of 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC with a delay of 35 ms (attempt = 1)
W0619 00:11:59.288486 25078 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 309a14ee0658485b9ea724d07b8cce20 consensus state has no leader
I0619 00:11:59.290561 25727 tablet_bootstrap.cc:427] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:59.291703 25727 ts_tablet_manager.cc:1045] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.098s	user 0.005s	sys 0.000s
I0619 00:11:59.281689 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:59.292809 25725 ts_tablet_manager.cc:1073] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.066s	user 0.003s	sys 0.001s
I0619 00:11:59.293215 25725 ts_tablet_manager.cc:1028] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:59.293821 25725 tablet_bootstrap.cc:427] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:59.294950 25727 raft_consensus.cc:325] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.295543 25727 raft_consensus.cc:351] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.295747 25727 raft_consensus.cc:628] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:59.296124 25727 consensus_queue.cc:227] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.298765 25724 tablet_bootstrap.cc:427] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:59.299100 25724 ts_tablet_manager.cc:1045] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.081s	user 0.005s	sys 0.000s
I0619 00:11:59.299516 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 2066 ms (attempt = 8)
W0619 00:11:59.299787 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
I0619 00:11:59.300034 25069 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 1050 ms (attempt = 7)
W0619 00:11:59.300289 25069 catalog_manager.cc:2910] Async tablet task 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:11:59.300686 25330 tablet_service.cc:759] Processing CreateTablet for tablet 9a38dedf214144739519839883b4ca59 (table=default.table_e5b8888e6fdb4ede8f40117d47639b8b [id=6324147fd0974be596a8c79042245c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.301303 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9a38dedf214144739519839883b4ca59. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.303140 25725 tablet_bootstrap.cc:578] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:59.305223 25598 ts_tablet_manager.cc:1173] T 536870e2d8d94dbba25bdd0df31bef17 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.311151 25599 tablet_service.cc:759] Processing CreateTablet for tablet 6064a4e9407f461e812c4f386dec72be (table=default.table_a06bcbc25e4f4009bb30ac5941a0f8d4 [id=1bd5615d11fc4c549fbecce081634a59]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.311734 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6064a4e9407f461e812c4f386dec72be. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.314748 25323 tablet_service.cc:759] Processing CreateTablet for tablet 536870e2d8d94dbba25bdd0df31bef17 (table=default.table_2784daa36b93413aa15c32837edf0490 [id=b5ff0ee44b064715b2fc34fcbb7ad5b5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.314858 25336 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.315659 25336 tablet_service.cc:800] Processing DeleteTablet for tablet b8bc8d4a970843258ab5e9530f425d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.316968 25067 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 519 ms (attempt = 6)
I0619 00:11:59.317251 25724 raft_consensus.cc:325] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.317930 25724 raft_consensus.cc:351] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.318148 25724 raft_consensus.cc:628] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:59.318856 25602 ts_tablet_manager.cc:1173] T 9a38dedf214144739519839883b4ca59 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.318892 25724 consensus_queue.cc:227] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.323268 25069 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 67 ms (attempt = 3)
W0619 00:11:59.323527 25069 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
W0619 00:11:59.317443 25067 catalog_manager.cc:2910] Async tablet task 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef consensus state has no leader
I0619 00:11:59.326534 25336 tablet_service.cc:759] Processing CreateTablet for tablet 9de47ddccf864f829fbc8c1968c891bd (table=default.table_ad726834d8e24054b8d217e0bc82629f [id=5516345f1dff43f7bfebafb0ed857b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.332659 25326 tablet_service.cc:759] Processing CreateTablet for tablet 6064a4e9407f461e812c4f386dec72be (table=default.table_a06bcbc25e4f4009bb30ac5941a0f8d4 [id=1bd5615d11fc4c549fbecce081634a59]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.333194 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 89 ms (attempt = 3)
W0619 00:11:59.333444 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.333822 25727 ts_tablet_manager.cc:1073] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.000s
I0619 00:11:59.303364 26266 raft_consensus.cc:457] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.336390 25081 master_service.cc:523] ReplaceTablet: received request to replace tablet 0c1f35568fe24c6c8cd2b88871b4520d from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:59.338300 25330 ts_tablet_manager.cc:1173] T 9a38dedf214144739519839883b4ca59 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.339483 25067 catalog_manager.cc:2891] Scheduling retry of 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC with a delay of 45 ms (attempt = 2)
I0619 00:11:59.339715 26272 raft_consensus.cc:457] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.322221 25599 ts_tablet_manager.cc:1173] T 6064a4e9407f461e812c4f386dec72be P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.328954 26267 raft_consensus.cc:457] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.340003 26272 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.955s (starting election)
I0619 00:11:59.340288 26272 raft_consensus.cc:479] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:59.339829 25067 catalog_manager.cc:2910] Async tablet task 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 309a14ee0658485b9ea724d07b8cce20 consensus state has no leader
I0619 00:11:59.341796 26272 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.342211 26272 leader_election.cc:243] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.346630 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6064a4e9407f461e812c4f386dec72be. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.347076 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 536870e2d8d94dbba25bdd0df31bef17. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.347869 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9de47ddccf864f829fbc8c1968c891bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.349122 25621 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:59.334199 26266 raft_consensus.cc:2732] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.030s (starting election)
W0619 00:11:59.351267 25290 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:11:59.351279 26266 raft_consensus.cc:479] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.352298 26266 leader_election.cc:243] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.352730 26266 leader_election.cc:243] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.354493 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f75b8a4f1fd845fbab6f83f59b7cfc58" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:59.355003 25598 tablet_service.cc:759] Processing CreateTablet for tablet 9de47ddccf864f829fbc8c1968c891bd (table=default.table_ad726834d8e24054b8d217e0bc82629f [id=5516345f1dff43f7bfebafb0ed857b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.355608 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9de47ddccf864f829fbc8c1968c891bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.356406 25354 raft_consensus.cc:2784] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.359971 25323 ts_tablet_manager.cc:1173] T 536870e2d8d94dbba25bdd0df31bef17 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.360014 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 2063 ms (attempt = 8)
W0619 00:11:59.360304 25071 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.345278 26267 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.228s (starting election)
W0619 00:11:59.363024 25553 leader_election.cc:306] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.363302 26267 raft_consensus.cc:479] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.364293 26267 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.364694 26267 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.365772 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:59.366417 25727 ts_tablet_manager.cc:1028] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:59.366813 25727 tablet_bootstrap.cc:427] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:59.366964 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 275 ms (attempt = 5)
W0619 00:11:59.367215 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.368523 26271 raft_consensus.cc:457] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.368866 26271 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.212s (starting election)
I0619 00:11:59.368784 25068 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 153 ms (attempt = 4)
W0619 00:11:59.369128 25068 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:59.369201 26271 raft_consensus.cc:479] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.369616 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 275 ms (attempt = 5)
W0619 00:11:59.370012 25067 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:59.370180 26271 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.370570 26271 leader_election.cc:243] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:59.367009 25553 leader_election.cc:306] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.372311 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:59.372871 25623 raft_consensus.cc:2206] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:11:59.372921 25336 ts_tablet_manager.cc:1173] T 9de47ddccf864f829fbc8c1968c891bd P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.374080 25069 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 539 ms (attempt = 6)
I0619 00:11:59.374023 25290 leader_election.cc:393] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
W0619 00:11:59.374349 25069 catalog_manager.cc:2910] Async tablet task 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
I0619 00:11:59.326117 25490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.376137 25490 raft_consensus.cc:2175] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:59.328109 26231 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.115s (election complete)
I0619 00:11:59.376761 26231 raft_consensus.cc:2487] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.343214 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d3c7114381c0423982b9ccbff120c2a2" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.373782 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1dd3f7afe05f49199b18220f9c11ccd2" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.377300 25489 raft_consensus.cc:2175] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:59.377276 25486 raft_consensus.cc:2175] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
W0619 00:11:59.329360 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 67041.9us
I0619 00:11:59.378374 25724 ts_tablet_manager.cc:1073] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.079s	user 0.004s	sys 0.000s
I0619 00:11:59.378434 26231 raft_consensus.cc:457] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.378684 26231 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.594s (starting election)
I0619 00:11:59.378866 25724 ts_tablet_manager.cc:1028] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:59.379221 25724 tablet_bootstrap.cc:427] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:59.378962 26231 raft_consensus.cc:479] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.379956 26231 leader_election.cc:243] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.381237 26231 leader_election.cc:243] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.382218 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41e052fdd92b4c3d9803fa43433e60d5" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:59.382971 25724 tablet_bootstrap.cc:578] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:11:59.384268 25419 leader_election.cc:306] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.384397 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9a221a1d9814ef99a0b4ccaa04d1017" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.385175 25558 leader_election.cc:393] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395763cf325d4ce19859830a7ad35c85 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:59.385283 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.385439 25558 leader_election.cc:270] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:11:59.292232 25736 ts_tablet_manager.cc:1382] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:11:59.385840 25558 raft_consensus.cc:2442] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:11:59.385900 25736 log.cc:986] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/d6c3c7f783cb43d59b46502261aeced3
I0619 00:11:59.386401 25736 ts_tablet_manager.cc:1403] T d6c3c7f783cb43d59b46502261aeced3 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:59.386903 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.387527 25423 leader_election.cc:379] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.387956 25423 leader_election.cc:270] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:59.388692 26231 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.806s (election complete)
I0619 00:11:59.388922 26231 raft_consensus.cc:2539] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:59.389149 26231 raft_consensus.cc:457] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:59.389353 26231 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.217s (starting election)
I0619 00:11:59.389459 25290 leader_election.cc:393] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:59.389555 26231 raft_consensus.cc:2784] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.389737 25290 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.390661 25290 leader_election.cc:393] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:11:59.390902 25290 leader_election.cc:270] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.391304 26272 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.650s (election complete)
I0619 00:11:59.391607 26272 raft_consensus.cc:2784] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.392110 25069 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 137 ms (attempt = 4)
W0619 00:11:59.392398 25069 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:59.394201 25326 ts_tablet_manager.cc:1173] T 6064a4e9407f461e812c4f386dec72be P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.394690 26231 raft_consensus.cc:479] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.396526 25741 ts_tablet_manager.cc:1382] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:59.396805 25741 log.cc:986] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:11:59.397014 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 2057 ms (attempt = 8)
I0619 00:11:59.397265 25741 ts_tablet_manager.cc:1403] T 0474be659e5e4a4cbfc181b8c812b1eb P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:59.397259 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:11:59.402686 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 4126 ms (attempt = 9)
W0619 00:11:59.402945 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:11:59.403262 25067 catalog_manager.cc:2891] Scheduling retry of 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC with a delay of 109 ms (attempt = 3)
W0619 00:11:59.403503 25067 catalog_manager.cc:2910] Async tablet task 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 309a14ee0658485b9ea724d07b8cce20 consensus state has no leader
I0619 00:11:59.406749 25602 tablet_service.cc:759] Processing CreateTablet for tablet 7656bfb303ab417391eece5c93e080bd (table=default.table_c5846407e88e4f36b378a06980f17f52 [id=d825926f52854d0f84b52351154a066a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.407107 25599 tablet_service.cc:759] Processing CreateTablet for tablet 551fd06ba0c74bae9a152c7c2ab989f4 (table=default.table_08c6a59c13db436bac657c6610d3ba2d [id=1cc3969f059d442eb594a9d4c719a8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.407374 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7656bfb303ab417391eece5c93e080bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.407685 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 551fd06ba0c74bae9a152c7c2ab989f4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.408574 25603 tablet_service.cc:759] Processing CreateTablet for tablet 8252076f40e1409681d92ba4cca0b0e6 (table=default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.409188 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8252076f40e1409681d92ba4cca0b0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.411139 25598 ts_tablet_manager.cc:1173] T 9de47ddccf864f829fbc8c1968c891bd P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.387909 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 353) took 3598ms. Request Metrics: {"tablet-delete.queue_time_us":3262656}
I0619 00:11:59.413561 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 354) took 3562ms. Request Metrics: {"tablet-delete.queue_time_us":3548461}
I0619 00:11:59.414162 25736 tablet_replica.cc:261] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:59.414407 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.415033 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.415185 26271 raft_consensus.cc:2732] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.212s (election complete)
I0619 00:11:59.415413 26271 raft_consensus.cc:2784] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.415416 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.415730 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.416148 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.416481 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.416857 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:11:59.417382 25736 raft_consensus.cc:2042] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.419191 25736 raft_consensus.cc:2069] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.419960 25290 leader_election.cc:393] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
I0619 00:11:59.420358 25290 leader_election.cc:270] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.421314 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 1056 ms (attempt = 7)
I0619 00:11:59.422950 25940 raft_consensus.cc:2732] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.060s (election complete)
I0619 00:11:59.423207 25940 raft_consensus.cc:2784] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.423615 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.424365 25725 tablet_bootstrap.cc:427] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:59.424746 25725 ts_tablet_manager.cc:1045] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.131s	user 0.006s	sys 0.000s
I0619 00:11:59.425114 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.426113 25725 raft_consensus.cc:325] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.434655 25940 raft_consensus.cc:2487] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:59.423562 25558 leader_election.cc:306] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.440757 25558 leader_election.cc:270] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.441135 25724 tablet_bootstrap.cc:427] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:59.441460 25724 ts_tablet_manager.cc:1045] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.062s	user 0.004s	sys 0.004s
I0619 00:11:59.442134 25603 ts_tablet_manager.cc:1173] T 8252076f40e1409681d92ba4cca0b0e6 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.442765 25621 raft_consensus.cc:1081] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:59.443907 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41e052fdd92b4c3d9803fa43433e60d5" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:59.444396 25616 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.120s (vote granted)
I0619 00:11:59.444618 25616 raft_consensus.cc:2245] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:59.441609 26267 raft_consensus.cc:2732] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.037s (election complete)
I0619 00:11:59.445191 26267 raft_consensus.cc:2487] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.424149 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 159 ms (attempt = 4)
W0619 00:11:59.445992 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:59.438402 25085 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 16 ms (attempt = 1)
W0619 00:11:59.446022 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
W0619 00:11:59.446243 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:59.443469 25078 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_6486a9f255374893bfb9f02f620c80da" }
I0619 00:11:59.445636 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_d74a1686828b4ee4b21231219c545181" }
I0619 00:11:59.444983 25082 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.447650 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.395591 26231 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.453517 25323 tablet_service.cc:759] Processing CreateTablet for tablet 8252076f40e1409681d92ba4cca0b0e6 (table=default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.426472 25336 tablet_service.cc:759] Processing CreateTablet for tablet 551fd06ba0c74bae9a152c7c2ab989f4 (table=default.table_08c6a59c13db436bac657c6610d3ba2d [id=1cc3969f059d442eb594a9d4c719a8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.454193 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8252076f40e1409681d92ba4cca0b0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.425802 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) successfully deleted
I0619 00:11:59.427021 26272 raft_consensus.cc:2487] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.458184 26231 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.460345 25725 raft_consensus.cc:351] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:11:59.458628 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 8ca3a0771b9e41eeb0d14d886f970159 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 8ca3a0771b9e41eeb0d14d886f970159 already in progress: creating tablet
I0619 00:11:59.462666 25071 catalog_manager.cc:2891] Scheduling retry of 8ca3a0771b9e41eeb0d14d886f970159 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 36 ms (attempt = 1)
I0619 00:11:59.463207 25069 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 78 ms (attempt = 2)
W0619 00:11:59.463472 25069 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:59.464718 25725 raft_consensus.cc:628] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:59.465050 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 525 ms (attempt = 6)
W0619 00:11:59.465288 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.435688 25736 ts_tablet_manager.cc:1369] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.465755 25600 tablet_service.cc:759] Processing CreateTablet for tablet 4492e3339ba149a5bbd2f581eea6b7ef (table=default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.466125 25078 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 18dbdb33a25e487ab87ecb85af31fd55
I0619 00:11:59.466416 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 18dbdb33a25e487ab87ecb85af31fd55 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.466570 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.466802 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 18dbdb33a25e487ab87ecb85af31fd55 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.467161 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 18dbdb33a25e487ab87ecb85af31fd55 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.467643 25598 tablet_service.cc:759] Processing CreateTablet for tablet f36f544f426845c98ac76ada27a4419e (table=default.table_cf1ddefe056a429fa21c2b9baf7236a9 [id=d523d6ade07547dfa1f27882a47542a4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.467911 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 92fc5418174246bba92169339f264d62
I0619 00:11:59.468189 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 92fc5418174246bba92169339f264d62 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.468448 25603 tablet_service.cc:759] Processing CreateTablet for tablet edfc7866c0974f139a0ac476cd19970f (table=default.table_a894c2fb24d74fc99270126b7197d3b1 [id=7b6597e467374b32a02f664f7d72be93]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.469060 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 92fc5418174246bba92169339f264d62 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.469424 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 92fc5418174246bba92169339f264d62 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.453680 25326 tablet_service.cc:759] Processing CreateTablet for tablet 7656bfb303ab417391eece5c93e080bd (table=default.table_c5846407e88e4f36b378a06980f17f52 [id=d825926f52854d0f84b52351154a066a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.469877 25725 consensus_queue.cc:227] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.470433 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7656bfb303ab417391eece5c93e080bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.471987 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 18dbdb33a25e487ab87ecb85af31fd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.472234 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 551fd06ba0c74bae9a152c7c2ab989f4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.472735 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 92fc5418174246bba92169339f264d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.431749 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 320) took 4522ms. Request Metrics: {"tablet-delete.queue_time_us":4317898}
I0619 00:11:59.474638 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4492e3339ba149a5bbd2f581eea6b7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.475124 25355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:11:59.477120 25419 leader_election.cc:306] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.478814 25725 ts_tablet_manager.cc:1073] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.054s	user 0.004s	sys 0.000s
I0619 00:11:59.481631 25087 catalog_manager.cc:3663] Got report from deleted tablet cbd67634f5704165b5bb948501697f0c (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) (replaced by tablet 507f7ed270af42ce8f404d55b5cae468): Sending delete request for this tablet
I0619 00:11:59.482399 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cbd67634f5704165b5bb948501697f0c on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 507f7ed270af42ce8f404d55b5cae468)
I0619 00:11:59.484475 25330 tablet_service.cc:759] Processing CreateTablet for tablet edfc7866c0974f139a0ac476cd19970f (table=default.table_a894c2fb24d74fc99270126b7197d3b1 [id=7b6597e467374b32a02f664f7d72be93]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.485067 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet edfc7866c0974f139a0ac476cd19970f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.486289 25350 raft_consensus.cc:1081] T 1dd3f7afe05f49199b18220f9c11ccd2 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 395763cf325d4ce19859830a7ad35c85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:59.474124 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 324) took 4512ms. Request Metrics: {"tablet-delete.queue_time_us":4511987}
I0619 00:11:59.471997 25730 ts_tablet_manager.cc:1382] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:59.487201 25730 log.cc:986] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/d3c7114381c0423982b9ccbff120c2a2
I0619 00:11:59.487867 26267 consensus_queue.cc:963] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:59.489020 25730 ts_tablet_manager.cc:1403] T d3c7114381c0423982b9ccbff120c2a2 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:59.489454 25350 raft_consensus.cc:1081] T 9d44681acacf4f04b6fcfbd4c9a18e87 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:59.490722 25725 ts_tablet_manager.cc:1028] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:59.491256 26231 consensus_queue.cc:963] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I0619 00:11:59.493239 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 325) took 4528ms. Request Metrics: {"tablet-delete.queue_time_us":4527580}
I0619 00:11:59.494571 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 329) took 4416ms. Request Metrics: {"tablet-delete.queue_time_us":4415155}
I0619 00:11:59.496945 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_33f663222f084af580067f0f88d85725" } new_table_name: "default.table_aad8b0f2529e4f288b86d31a9769d357" alter_external_catalogs: true
I0619 00:11:59.499855 25741 tablet_replica.cc:261] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:59.491333 25725 tablet_bootstrap.cc:427] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:59.503036 25725 tablet_bootstrap.cc:578] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:59.497190 25727 tablet_bootstrap.cc:578] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:59.505085 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:59.505316 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:59.505501 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:59.515625 25067 catalog_manager.cc:2891] Scheduling retry of 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC with a delay of 134 ms (attempt = 4)
I0619 00:11:59.521543 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:59.516604 25331 tablet_service.cc:759] Processing CreateTablet for tablet f36f544f426845c98ac76ada27a4419e (table=default.table_cf1ddefe056a429fa21c2b9baf7236a9 [id=d523d6ade07547dfa1f27882a47542a4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.511771 25323 ts_tablet_manager.cc:1173] T 8252076f40e1409681d92ba4cca0b0e6 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.509500 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f75b8a4f1fd845fbab6f83f59b7cfc58" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.510898 25469 tablet_service.cc:759] Processing CreateTablet for tablet 9a38dedf214144739519839883b4ca59 (table=default.table_e5b8888e6fdb4ede8f40117d47639b8b [id=6324147fd0974be596a8c79042245c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.522745 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f36f544f426845c98ac76ada27a4419e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.522976 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9a38dedf214144739519839883b4ca59. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.512346 25465 tablet_service.cc:759] Processing CreateTablet for tablet 536870e2d8d94dbba25bdd0df31bef17 (table=default.table_2784daa36b93413aa15c32837edf0490 [id=b5ff0ee44b064715b2fc34fcbb7ad5b5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.513483 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.523497 25486 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.190s (vote granted)
I0619 00:11:59.523751 25486 raft_consensus.cc:2245] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
I0619 00:11:59.514420 25467 tablet_service.cc:759] Processing CreateTablet for tablet 9de47ddccf864f829fbc8c1968c891bd (table=default.table_ad726834d8e24054b8d217e0bc82629f [id=5516345f1dff43f7bfebafb0ed857b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.524487 25465 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 536870e2d8d94dbba25bdd0df31bef17. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.512369 26145 consensus_queue.cc:963] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:59.527736 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9de47ddccf864f829fbc8c1968c891bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.531450 25331 ts_tablet_manager.cc:1173] T f36f544f426845c98ac76ada27a4419e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.533112 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:59.533244 25469 ts_tablet_manager.cc:1173] T 9a38dedf214144739519839883b4ca59 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.533332 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:59.533520 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(0a33940e5347489993ab4679e9aeafde)
I0619 00:11:59.533756 25741 raft_consensus.cc:2042] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.534082 25741 raft_consensus.cc:2069] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.544428 25465 ts_tablet_manager.cc:1173] T 536870e2d8d94dbba25bdd0df31bef17 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.502364 26271 raft_consensus.cc:2487] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.488405 25601 tablet_service.cc:800] Processing DeleteTablet for tablet cbd67634f5704165b5bb948501697f0c with delete_type TABLET_DATA_DELETED (replaced by tablet 507f7ed270af42ce8f404d55b5cae468) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.504048 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet edfc7866c0974f139a0ac476cd19970f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.515116 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:11:59.545068 25616 raft_consensus.cc:2784] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.545315 25616 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.850s (vote granted)
I0619 00:11:59.521096 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:59.529858 25069 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 270 ms (attempt = 5)
W0619 00:11:59.547060 25069 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:59.547482 25069 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 88 ms (attempt = 3)
W0619 00:11:59.547742 25069 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:59.529595 25068 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC with a delay of 285 ms (attempt = 5)
W0619 00:11:59.548439 25068 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:59.548759 25078 master_service.cc:523] ReplaceTablet: received request to replace tablet d62715e230154e2abe126ef97feec7ce from {username='jenkins'} at 127.24.119.62:54390
I0619 00:11:59.549777 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 2048 ms (attempt = 8)
W0619 00:11:59.550060 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:59.550823 25078 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_4eb928c7853a4940ba9a83ba7f30a181"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.551704 25078 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4eb928c7853a4940ba9a83ba7f30a181 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.536303 25087 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 44 ms (attempt = 1)
I0619 00:11:59.552783 25082 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_18b6ad6c2b1142c4a307bb3a4d56f328"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.552820 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
W0619 00:11:59.553648 25082 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_18b6ad6c2b1142c4a307bb3a4d56f328 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.554422 25326 ts_tablet_manager.cc:1173] T 7656bfb303ab417391eece5c93e080bd P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.558605 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f36f544f426845c98ac76ada27a4419e. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:11:59.554518 25067 catalog_manager.cc:2910] Async tablet task 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 309a14ee0658485b9ea724d07b8cce20 consensus state has no leader
I0619 00:11:59.560519 25616 raft_consensus.cc:2245] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:11:59.561728 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 256) took 4000ms. Request Metrics: {"tablet-delete.queue_time_us":3602695}
I0619 00:11:59.562708 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet d3c7114381c0423982b9ccbff120c2a2 (table default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]) successfully deleted
W0619 00:11:59.572111 25558 leader_election.cc:306] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.572355 25558 leader_election.cc:270] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.573694 26267 raft_consensus.cc:457] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.573777 26223 raft_consensus.cc:2732] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.609s (election complete)
I0619 00:11:59.573982 26267 raft_consensus.cc:2732] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.961s (starting election)
I0619 00:11:59.574035 26223 raft_consensus.cc:2487] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.574234 26267 raft_consensus.cc:479] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.575237 26267 leader_election.cc:243] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.552673 25736 ts_tablet_manager.cc:1382] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:59.575234 25558 leader_election.cc:379] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.575883 25558 leader_election.cc:270] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:11:59.575871 25736 log.cc:986] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/d3c7114381c0423982b9ccbff120c2a2
I0619 00:11:59.576056 26267 leader_election.cc:243] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.573087 25462 tablet_service.cc:759] Processing CreateTablet for tablet 551fd06ba0c74bae9a152c7c2ab989f4 (table=default.table_08c6a59c13db436bac657c6610d3ba2d [id=1cc3969f059d442eb594a9d4c719a8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.576429 25736 ts_tablet_manager.cc:1403] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:59.576656 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 551fd06ba0c74bae9a152c7c2ab989f4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.577378 25467 ts_tablet_manager.cc:1173] T 9de47ddccf864f829fbc8c1968c891bd P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.577282 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 355) took 3699ms. Request Metrics: {"tablet-delete.queue_time_us":3535428}
I0619 00:11:59.561511 25725 tablet_bootstrap.cc:427] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:59.577949 25486 raft_consensus.cc:1081] T 1dd3f7afe05f49199b18220f9c11ccd2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer 395763cf325d4ce19859830a7ad35c85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:11:59.578269 25725 ts_tablet_manager.cc:1045] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.087s	user 0.004s	sys 0.001s
I0619 00:11:59.579805 26223 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.242s (election complete)
I0619 00:11:59.580035 26223 raft_consensus.cc:2539] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:59.580235 26223 raft_consensus.cc:457] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:59.580430 26223 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.771s (starting election)
I0619 00:11:59.580621 26223 raft_consensus.cc:2784] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:59.582504 25067 catalog_manager.cc:2910] Async tablet task 92fc5418174246bba92169339f264d62 (table default.table_d74a1686828b4ee4b21231219c545181 [id=f0a9a632ab31474aaf63e7787bb23f06]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 92fc5418174246bba92169339f264d62 consensus state has no leader
I0619 00:11:59.579603 25725 raft_consensus.cc:325] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.584487 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 0474be659e5e4a4cbfc181b8c812b1eb (table default.table_e248c37d574c4073ad57f7b09e7bf793 [id=ef686082956f4a898d38d2526484ec1b]) successfully deleted
I0619 00:11:59.584779 25725 raft_consensus.cc:351] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.584969 25725 raft_consensus.cc:628] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:59.563215 25724 raft_consensus.cc:325] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.585391 25725 consensus_queue.cc:227] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:59.586401 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet d0624684b6b04ba3be260b2705dc00e1 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet d0624684b6b04ba3be260b2705dc00e1 already in progress: creating tablet
I0619 00:11:59.586832 25462 ts_tablet_manager.cc:1173] T 551fd06ba0c74bae9a152c7c2ab989f4 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.587321 26223 raft_consensus.cc:479] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.588405 26223 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.588848 26223 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.585551 25724 raft_consensus.cc:351] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.589433 25724 raft_consensus.cc:628] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:59.590471 25067 catalog_manager.cc:2891] Scheduling retry of d0624684b6b04ba3be260b2705dc00e1 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 60 ms (attempt = 1)
W0619 00:11:59.591526 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet f1b566b2b58f4955aa25ae990a4a8a1a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f1b566b2b58f4955aa25ae990a4a8a1a already in progress: creating tablet
I0619 00:11:59.591548 25724 consensus_queue.cc:227] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.573261 25469 tablet_service.cc:759] Processing CreateTablet for tablet 6064a4e9407f461e812c4f386dec72be (table=default.table_a06bcbc25e4f4009bb30ac5941a0f8d4 [id=1bd5615d11fc4c549fbecce081634a59]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.596042 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6064a4e9407f461e812c4f386dec72be. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.596092 25068 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 1042 ms (attempt = 7)
W0619 00:11:59.596371 25068 catalog_manager.cc:2910] Async tablet task 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:11:59.574993 25602 ts_tablet_manager.cc:1173] T 7656bfb303ab417391eece5c93e080bd P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.597848 25725 ts_tablet_manager.cc:1073] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.001s
I0619 00:11:59.598426 25725 ts_tablet_manager.cc:1028] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:59.599050 25725 tablet_bootstrap.cc:427] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:11:59.592031 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 60 ms (attempt = 1)
I0619 00:11:59.599139 25599 ts_tablet_manager.cc:1173] T 551fd06ba0c74bae9a152c7c2ab989f4 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.604375 25464 tablet_service.cc:759] Processing CreateTablet for tablet 7656bfb303ab417391eece5c93e080bd (table=default.table_c5846407e88e4f36b378a06980f17f52 [id=d825926f52854d0f84b52351154a066a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.605012 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7656bfb303ab417391eece5c93e080bd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.567101 25331 tablet_service.cc:759] Processing CreateTablet for tablet 4492e3339ba149a5bbd2f581eea6b7ef (table=default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.568382 25727 tablet_bootstrap.cc:427] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:59.605540 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 278 ms (attempt = 5)
I0619 00:11:59.605834 25727 ts_tablet_manager.cc:1045] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.239s	user 0.006s	sys 0.000s
W0619 00:11:59.605823 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.607233 25727 raft_consensus.cc:325] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.607863 25727 raft_consensus.cc:351] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.608039 25727 raft_consensus.cc:628] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:59.590435 25325 tablet_service.cc:800] Processing DeleteTablet for tablet 92fc5418174246bba92169339f264d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.577955 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 357) took 3650ms. Request Metrics: {"tablet-delete.queue_time_us":3649744}
I0619 00:11:59.589902 25323 tablet_service.cc:800] Processing DeleteTablet for tablet 18dbdb33a25e487ab87ecb85af31fd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.591153 25741 ts_tablet_manager.cc:1369] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:11:59.609531 25071 catalog_manager.cc:2910] Async tablet task 4c92d341c4d94e4987524a056e703310 (table default.table_718ed0cda54b4ab8b71539f7f044d28f [id=01bad0c1690944cca1aaaffd4d454de5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c92d341c4d94e4987524a056e703310 consensus state has no leader
I0619 00:11:59.609283 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 358) took 3676ms. Request Metrics: {"tablet-delete.queue_time_us":3676190}
I0619 00:11:59.608465 25727 consensus_queue.cc:227] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.615713 25727 ts_tablet_manager.cc:1073] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.001s
I0619 00:11:59.616634 25336 ts_tablet_manager.cc:1173] T 551fd06ba0c74bae9a152c7c2ab989f4 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.616631 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 359) took 3548ms. Request Metrics: {"tablet-delete.queue_time_us":3529532}
W0619 00:11:59.602447 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4a88e394b199431e8b794a1b1aec847f because the tablet was not found. No further retry: Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:59.618511 25356 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8e3a356933e749b9a4ade476ec30f475" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:59.619215 25353 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:11:59.621186 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4492e3339ba149a5bbd2f581eea6b7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.625469 25067 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 2052 ms (attempt = 8)
I0619 00:11:59.623459 25330 ts_tablet_manager.cc:1173] T edfc7866c0974f139a0ac476cd19970f P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.624150 25423 leader_election.cc:393] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10bda0b4856f4441b5aed24cdc4764a8 in current term 1: Already voted for candidate 395763cf325d4ce19859830a7ad35c85 in this term.
W0619 00:11:59.626444 25067 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:59.632889 26286 tablet.cc:1200] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:59.633399 25067 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 54 ms (attempt = 2)
W0619 00:11:59.633961 25067 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:59.636790 25069 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 175 ms (attempt = 4)
W0619 00:11:59.637046 25069 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:11:59.638833 25462 tablet_service.cc:759] Processing CreateTablet for tablet 8252076f40e1409681d92ba4cca0b0e6 (table=default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.639427 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8252076f40e1409681d92ba4cca0b0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.640497 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8e3a356933e749b9a4ade476ec30f475" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.641178 26223 consensus_queue.cc:963] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:11:59.641679 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet d3c7114381c0423982b9ccbff120c2a2 (table default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]) successfully deleted
I0619 00:11:59.642647 25467 tablet_service.cc:759] Processing CreateTablet for tablet edfc7866c0974f139a0ac476cd19970f (table=default.table_a894c2fb24d74fc99270126b7197d3b1 [id=7b6597e467374b32a02f664f7d72be93]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.643199 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet edfc7866c0974f139a0ac476cd19970f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.643681 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 529 ms (attempt = 6)
W0619 00:11:59.643924 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.645548 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 363) took 3507ms. Request Metrics: {"tablet-delete.queue_time_us":3478530}
I0619 00:11:59.635948 25727 ts_tablet_manager.cc:1028] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
W0619 00:11:59.643359 25402 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.24.119.61:46325 timed out after 1.000s (SENT)
I0619 00:11:59.648401 25727 tablet_bootstrap.cc:427] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:59.650631 25727 tablet_bootstrap.cc:578] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:59.648069 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 551 ms (attempt = 6)
W0619 00:11:59.651808 25067 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:11:59.652601 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:11:59.654942 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 364) took 3516ms. Request Metrics: {"tablet-delete.queue_time_us":3515345}
I0619 00:11:59.655386 25402 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:59.655509 25736 tablet_replica.cc:261] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:59.656934 25331 ts_tablet_manager.cc:1173] T 4492e3339ba149a5bbd2f581eea6b7ef P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.658195 26271 raft_consensus.cc:457] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.658404 26271 raft_consensus.cc:2732] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.698s (starting election)
I0619 00:11:59.658654 26271 raft_consensus.cc:479] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.659574 26271 leader_election.cc:243] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.597828 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 255) took 4037ms. Request Metrics: {"tablet-delete.queue_time_us":4017415}
I0619 00:11:59.660199 25730 tablet_replica.cc:261] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:59.660229 25940 raft_consensus.cc:457] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.660459 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(f75b8a4f1fd845fbab6f83f59b7cfc58)
I0619 00:11:59.660483 25940 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.965s (starting election)
W0619 00:11:59.660548 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 8ca3a0771b9e41eeb0d14d886f970159 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 8ca3a0771b9e41eeb0d14d886f970159 already in progress: creating tablet
W0619 00:11:59.660548 25553 leader_election.cc:306] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.660825 25071 catalog_manager.cc:2891] Scheduling retry of 8ca3a0771b9e41eeb0d14d886f970159 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 35 ms (attempt = 1)
I0619 00:11:59.660758 25940 raft_consensus.cc:479] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.659968 26271 leader_election.cc:243] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
W0619 00:11:59.661260 25553 leader_election.cc:306] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.660696 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(f75b8a4f1fd845fbab6f83f59b7cfc58)
I0619 00:11:59.661664 25940 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.661706 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(f75b8a4f1fd845fbab6f83f59b7cfc58)
I0619 00:11:59.661967 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(f75b8a4f1fd845fbab6f83f59b7cfc58)
I0619 00:11:59.662051 25940 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.662200 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(f75b8a4f1fd845fbab6f83f59b7cfc58)
I0619 00:11:59.662413 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(f75b8a4f1fd845fbab6f83f59b7cfc58)
I0619 00:11:59.662607 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(f75b8a4f1fd845fbab6f83f59b7cfc58)
I0619 00:11:59.662830 25730 raft_consensus.cc:2042] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.663110 25730 raft_consensus.cc:2069] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.665097 25730 ts_tablet_manager.cc:1369] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.666575 25080 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_da957178ddc241ed8e414551ec468a12" } new_table_name: "default.table_94f6d4fdfe6c462cb13a00dbdef20cd3" alter_external_catalogs: true
I0619 00:11:59.666483 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_defc57494cc44555be460e4b77d34fe8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.667431 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_defc57494cc44555be460e4b77d34fe8 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.667479 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_32abea053929403b9fd5b7a0b6189479"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.668359 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_32abea053929403b9fd5b7a0b6189479 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.669306 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:59.671919 25083 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_06ca188d31e6404585c64d19afe3dc8a" }
I0619 00:11:59.672389 25086 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_297b8bff60264a579f026fbd82396f53" }
I0619 00:11:59.674847 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.675082 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.675271 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.675470 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.675935 25598 ts_tablet_manager.cc:1173] T f36f544f426845c98ac76ada27a4419e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.676511 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc7686daae274e0dba87703fe6924805" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.677019 25486 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.931s (vote granted)
I0619 00:11:59.677232 25486 raft_consensus.cc:2245] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:11:59.678489 25079 catalog_manager.cc:3663] Got report from deleted tablet b50379774e4d4fdfb823ab2dc919db30 (table default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb): Sending delete request for this tablet
I0619 00:11:59.679189 25079 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b50379774e4d4fdfb823ab2dc919db30 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb)
I0619 00:11:59.680064 25724 ts_tablet_manager.cc:1073] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.238s	user 0.004s	sys 0.003s
I0619 00:11:59.685166 25724 ts_tablet_manager.cc:1028] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:59.688838 25081 catalog_manager.cc:3663] Got report from deleted tablet 1cfd335c18c047fba2facc5194807abb (table default.table_2af4c8dc619643729ba189a0dd8cd15f [id=ad26a268092e4cc4a44e6845dc511880]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.689435 25724 tablet_bootstrap.cc:427] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:59.689493 25081 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1cfd335c18c047fba2facc5194807abb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.690524 26223 raft_consensus.cc:457] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.690872 26223 raft_consensus.cc:2732] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.002s (starting election)
I0619 00:11:59.691136 26223 raft_consensus.cc:479] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.692203 26223 leader_election.cc:243] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.693162 26223 leader_election.cc:243] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.701004 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cbd67634f5704165b5bb948501697f0c" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:59.702811 25536 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.24.119.61:46325 timed out after 1.000s (SENT)
I0619 00:11:59.703604 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.703794 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:59.703915 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.704180 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(30a9d9fda8d34caeb8242c372439321b)
W0619 00:11:59.702630 25553 leader_election.cc:306] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.704393 25736 raft_consensus.cc:2042] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:11:59.704659 25736 consensus_peers.cc:456] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:11:59.704926 25736 consensus_peers.cc:456] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:11:59.705349 25736 raft_consensus.cc:2069] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.705747 25725 tablet_bootstrap.cc:578] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:59.708328 25600 ts_tablet_manager.cc:1173] T 4492e3339ba149a5bbd2f581eea6b7ef P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.710731 25724 tablet_bootstrap.cc:578] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:59.712776 25741 ts_tablet_manager.cc:1382] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:11:59.714237 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.718714 25080 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_94f6d4fdfe6c462cb13a00dbdef20cd3 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 59 ms (attempt = 1)
W0619 00:11:59.718981 25080 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:11:59.720026 25465 tablet_service.cc:759] Processing CreateTablet for tablet f36f544f426845c98ac76ada27a4419e (table=default.table_cf1ddefe056a429fa21c2b9baf7236a9 [id=d523d6ade07547dfa1f27882a47542a4]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.720600 25465 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f36f544f426845c98ac76ada27a4419e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.719027 25741 log.cc:986] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/0a33940e5347489993ab4679e9aeafde
I0619 00:11:59.721433 25741 ts_tablet_manager.cc:1403] T 0a33940e5347489993ab4679e9aeafde P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
W0619 00:11:59.723526 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a284677c4b7f46e18e25edee6024e3eb because the tablet was not found. No further retry: Not found: Tablet not found: a284677c4b7f46e18e25edee6024e3eb
W0619 00:11:59.723981 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 86959e7dda374ba9ac23d3c1fd30d993 because the tablet was not found. No further retry: Not found: Tablet not found: 86959e7dda374ba9ac23d3c1fd30d993
W0619 00:11:59.724407 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 54adb588955b45b79de9752f254728ea because the tablet was not found. No further retry: Not found: Tablet not found: 54adb588955b45b79de9752f254728ea
I0619 00:11:59.726145 25603 ts_tablet_manager.cc:1173] T edfc7866c0974f139a0ac476cd19970f P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.727639 25736 ts_tablet_manager.cc:1369] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.728775 26285 tablet.cc:1200] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:11:59.735795 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:59.735303 25067 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 112 ms (attempt = 3)
I0619 00:11:59.736208 25486 raft_consensus.cc:2784] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:11:59.736389 25067 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:11:59.736448 25486 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.216s (vote granted)
I0619 00:11:59.736747 25067 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC with a delay of 1028 ms (attempt = 7)
W0619 00:11:59.736982 25067 catalog_manager.cc:2910] Async tablet task 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
W0619 00:11:59.739969 25558 leader_election.cc:306] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.740255 25558 leader_election.cc:270] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.740867 26267 raft_consensus.cc:2732] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.882s (election complete)
I0619 00:11:59.741096 26267 raft_consensus.cc:2487] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.742768 25486 raft_consensus.cc:2245] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 1.
I0619 00:11:59.745199 25469 ts_tablet_manager.cc:1173] T 6064a4e9407f461e812c4f386dec72be P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.748844 25080 catalog_manager.cc:3663] Got report from deleted tablet b50379774e4d4fdfb823ab2dc919db30 (table default.table_0075a29451044e27990e84f7ae19482d [id=ee982ffd0ecb448792682763ae6244cd]) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb): Sending delete request for this tablet
I0619 00:11:59.749148 25080 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.750015 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b50379774e4d4fdfb823ab2dc919db30 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb)
W0619 00:11:59.750069 26247 lock_manager.cc:368] Waited 1 seconds to obtain row lock on key \x82baf80fd968c44e679987eb4ebe4473de cur holder: 0x7b4c000a2dc0
I0619 00:11:59.750396 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.752770 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 18dbdb33a25e487ab87ecb85af31fd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.753696 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 1c4c69294cb945938004e5b9a0550e72
I0619 00:11:59.753875 25083 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet fe93c5dd62d94bdda099b9131482c033
I0619 00:11:59.753986 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1c4c69294cb945938004e5b9a0550e72 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.754153 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fe93c5dd62d94bdda099b9131482c033 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.754415 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1c4c69294cb945938004e5b9a0550e72 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.754541 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fe93c5dd62d94bdda099b9131482c033 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.754814 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1c4c69294cb945938004e5b9a0550e72 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.754972 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fe93c5dd62d94bdda099b9131482c033 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:11:59.755755 25724 tablet_bootstrap.cc:427] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:59.756124 25724 ts_tablet_manager.cc:1045] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.067s	user 0.003s	sys 0.002s
I0619 00:11:59.758303 25536 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
I0619 00:11:59.758754 25080 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.759438 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
W0619 00:11:59.760607 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 8e3a356933e749b9a4ade476ec30f475 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 8e3a356933e749b9a4ade476ec30f475 already in progress: creating tablet
I0619 00:11:59.760895 25071 catalog_manager.cc:2891] Scheduling retry of 8e3a356933e749b9a4ade476ec30f475 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 44 ms (attempt = 1)
I0619 00:11:59.761117 25603 tablet_service.cc:800] Processing DeleteTablet for tablet b50379774e4d4fdfb823ab2dc919db30 with delete_type TABLET_DATA_DELETED (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:59.762413 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3e7c4b61f5314e40ad71371155a50b05 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3e7c4b61f5314e40ad71371155a50b05 already in progress: creating tablet
I0619 00:11:59.762697 25071 catalog_manager.cc:2891] Scheduling retry of 3e7c4b61f5314e40ad71371155a50b05 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 16 ms (attempt = 1)
I0619 00:11:59.763236 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.763665 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 1c4c69294cb945938004e5b9a0550e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.763829 25469 tablet_service.cc:759] Processing CreateTablet for tablet 4492e3339ba149a5bbd2f581eea6b7ef (table=default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.764194 25600 tablet_service.cc:800] Processing DeleteTablet for tablet fe93c5dd62d94bdda099b9131482c033 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.764463 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4492e3339ba149a5bbd2f581eea6b7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.764524 25330 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.764732 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 0a7470bfeab041f9895e491231b57eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:11:59.765563 25290 leader_election.cc:306] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 54373917567a479d93dc8a3fd006d2fe
I0619 00:11:59.765175 26285 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.24.119.2:51757 (request call id 478) took 1171ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":163811,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":814,"prepare.run_wall_time_us":815,"raft.queue_time_us":1569,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":485,"replication_time_us":969927,"thread_start_us":169328,"threads_started":1}]]}
W0619 00:11:59.767930 25290 leader_election.cc:306] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: 54373917567a479d93dc8a3fd006d2fe
I0619 00:11:59.768195 25290 leader_election.cc:270] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.768636 25940 raft_consensus.cc:2732] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.440s (election complete)
I0619 00:11:59.768836 25940 raft_consensus.cc:2487] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:11:59.769757 25290 leader_election.cc:306] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: fc7686daae274e0dba87703fe6924805
I0619 00:11:59.770205 25290 leader_election.cc:379] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.772845 25465 ts_tablet_manager.cc:1173] T f36f544f426845c98ac76ada27a4419e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.773820 25725 tablet_bootstrap.cc:427] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:11:59.774399 25725 ts_tablet_manager.cc:1045] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.175s	user 0.002s	sys 0.006s
I0619 00:11:59.737296 25067 catalog_manager.cc:2891] Scheduling retry of 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC with a delay of 2068 ms (attempt = 8)
W0619 00:11:59.776036 25067 catalog_manager.cc:2910] Async tablet task 507f7ed270af42ce8f404d55b5cae468 (table default.table_b5b52789c2e347ccb0587438d7002a6b [id=2b0d752c696449e899310b57d8c7572e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 507f7ed270af42ce8f404d55b5cae468 consensus state has no leader
I0619 00:11:59.776145 25558 leader_election.cc:379] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.776415 25558 leader_election.cc:270] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
W0619 00:11:59.776386 25423 leader_election.cc:306] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:11:59.776471 25067 catalog_manager.cc:2891] Scheduling retry of 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC with a delay of 261 ms (attempt = 5)
I0619 00:11:59.776667 25423 leader_election.cc:270] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
W0619 00:11:59.776890 25067 catalog_manager.cc:2910] Async tablet task 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 309a14ee0658485b9ea724d07b8cce20 consensus state has no leader
W0619 00:11:59.777066 25423 raft_consensus.cc:2442] T d3c7114381c0423982b9ccbff120c2a2 P 10bda0b4856f4441b5aed24cdc4764a8: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:11:59.777086 26223 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.595s (election complete)
I0619 00:11:59.777302 26223 raft_consensus.cc:2539] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:59.777578 26223 raft_consensus.cc:593] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Becoming Leader. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Running, Role: LEADER
I0619 00:11:59.777632 25465 tablet_service.cc:800] Processing DeleteTablet for tablet 92fc5418174246bba92169339f264d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.778354 25598 tablet_service.cc:800] Processing DeleteTablet for tablet b50379774e4d4fdfb823ab2dc919db30 with delete_type TABLET_DATA_DELETED (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.778023 26223 consensus_queue.cc:204] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.780622 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7c4b61f5314e40ad71371155a50b05 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:59.781278 25068 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_94f6d4fdfe6c462cb13a00dbdef20cd3 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 41 ms (attempt = 2)
W0619 00:11:59.781540 25068 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_94f6d4fdfe6c462cb13a00dbdef20cd3 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:11:59.785946 25290 leader_election.cc:270] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:11:59.786172 25068 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:11:59.786465 25940 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.177s (election complete)
I0619 00:11:59.786711 25940 raft_consensus.cc:2539] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:11:59.786901 25940 raft_consensus.cc:457] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:11:59.787078 25940 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.703s (starting election)
I0619 00:11:59.787261 25940 raft_consensus.cc:2784] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.757776 25724 raft_consensus.cc:325] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.774368 25331 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.789723 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 1c4c69294cb945938004e5b9a0550e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.789873 25724 raft_consensus.cc:351] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.792984 25724 raft_consensus.cc:628] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:59.788274 25725 raft_consensus.cc:325] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.795478 25730 ts_tablet_manager.cc:1382] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:11:59.723588 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 330) took 4636ms. Request Metrics: {"tablet-delete.queue_time_us":4411501}
I0619 00:11:59.796166 25725 raft_consensus.cc:351] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.799762 25940 raft_consensus.cc:479] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.801918 25940 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.802287 25940 leader_election.cc:243] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.798837 25725 raft_consensus.cc:628] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:11:59.803756 25730 log.cc:986] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/f75b8a4f1fd845fbab6f83f59b7cfc58
I0619 00:11:59.799069 25724 consensus_queue.cc:227] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.805856 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
I0619 00:11:59.796516 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 328) took 4725ms. Request Metrics: {"tablet-delete.queue_time_us":4682159}
I0619 00:11:59.806957 25724 ts_tablet_manager.cc:1073] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.051s	user 0.004s	sys 0.000s
I0619 00:11:59.808600 25724 ts_tablet_manager.cc:1028] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:11:59.808848 26145 raft_consensus.cc:457] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.809098 25724 tablet_bootstrap.cc:427] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:11:59.809106 26145 raft_consensus.cc:2732] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.784s (starting election)
I0619 00:11:59.809352 26145 raft_consensus.cc:479] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.810434 26145 leader_election.cc:243] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.811193 25465 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.811398 25724 tablet_bootstrap.cc:578] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:11:59.812528 25353 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6040e369e0fb445fb993d12dd883039b" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:11:59.813402 25069 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 256 ms (attempt = 5)
W0619 00:11:59.813701 25069 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
W0619 00:11:59.815464 25419 leader_election.cc:306] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.805316 25725 consensus_queue.cc:227] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.816993 26285 tablet.cc:1195] T 9e5e691e6d374cd4bba1f18d8c821723 P 10bda0b4856f4441b5aed24cdc4764a8: Already running schema version 1 got alter request for version 1
I0619 00:11:59.817023 25730 ts_tablet_manager.cc:1403] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:11:59.817885 26285 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.24.119.2:51757 (request call id 485) took 1151ms. Request Metrics: {"spinlock_wait_cycles":8241024,"child_traces":[["txn",{"apply.queue_time_us":63,"prepare.queue_time_us":82,"prepare.run_cpu_time_us":84982,"prepare.run_wall_time_us":1059947,"raft.queue_time_us":143,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":2616,"replication_time_us":50997}]]}
I0619 00:11:59.818408 25069 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 541 ms (attempt = 6)
W0619 00:11:59.818691 25069 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:11:59.819062 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 257) took 4202ms. Request Metrics: {"tablet-delete.queue_time_us":4010236}
I0619 00:11:59.819806 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet f75b8a4f1fd845fbab6f83f59b7cfc58 (table default.table_fa53e172a2a34136a5a6efb8b5c39368 [id=779e902dec1a4701a176d545a35e9978]) successfully deleted
I0619 00:11:59.823833 25068 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_94f6d4fdfe6c462cb13a00dbdef20cd3 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 95 ms (attempt = 3)
W0619 00:11:59.824095 25068 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_94f6d4fdfe6c462cb13a00dbdef20cd3 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:11:59.824131 26286 tablet.cc:1195] T 9e5e691e6d374cd4bba1f18d8c821723 P c007a00e65ad41a79da1a0c1050c55b4: Already running schema version 1 got alter request for version 1
I0619 00:11:59.826253 25331 tablet_service.cc:800] Processing DeleteTablet for tablet fe93c5dd62d94bdda099b9131482c033 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.827852 25087 catalog_manager.cc:3663] Got report from deleted tablet d3c7114381c0423982b9ccbff120c2a2 (table default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:11:59.828161 25087 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.828433 25725 ts_tablet_manager.cc:1073] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.054s	user 0.004s	sys 0.000s
I0619 00:11:59.828431 25087 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.828867 25087 catalog_manager.cc:3663] Got report from deleted tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:59.829135 25087 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:59.830770 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d3c7114381c0423982b9ccbff120c2a2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:59.831127 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:59.831472 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.831843 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:59.832372 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:59.834251 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 0a33940e5347489993ab4679e9aeafde (table default.table_2e052476ba2c4ab08ef5774928db6084 [id=806774dd321f4cbcb6f709a3d06cf72d]) successfully deleted
W0619 00:11:59.834841 25068 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
W0619 00:11:59.835654 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 4a88e394b199431e8b794a1b1aec847f because the tablet was not found. No further retry: Not found: Tablet not found: 4a88e394b199431e8b794a1b1aec847f
I0619 00:11:59.775708 25467 ts_tablet_manager.cc:1173] T edfc7866c0974f139a0ac476cd19970f P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.836951 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_08c6a59c13db436bac657c6610d3ba2d" } new_table_name: "default.table_7d4d48d8304c46b5a13a0576fc34d663" alter_external_catalogs: true
I0619 00:11:59.839347 25469 ts_tablet_manager.cc:1173] T 4492e3339ba149a5bbd2f581eea6b7ef P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.839597 25464 ts_tablet_manager.cc:1173] T 7656bfb303ab417391eece5c93e080bd P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.840095 26145 leader_election.cc:243] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.840198 25462 ts_tablet_manager.cc:1173] T 8252076f40e1409681d92ba4cca0b0e6 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.841413 25080 master_service.cc:523] ReplaceTablet: received request to replace tablet f2d15bd8d6ce460c8422625ba060c6e0 from {username='jenkins'} at 127.24.119.62:54390
W0619 00:11:59.832840 25290 leader_election.cc:306] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: fc7686daae274e0dba87703fe6924805
I0619 00:11:59.842989 25331 tablet_service.cc:800] Processing DeleteTablet for tablet d3c7114381c0423982b9ccbff120c2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.846259 25736 ts_tablet_manager.cc:1382] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
W0619 00:11:59.847810 25668 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:46325: Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.24.119.61:46325 timed out after 1.000s (SENT)
I0619 00:11:59.849858 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 261) took 4007ms. Request Metrics: {"tablet-delete.queue_time_us":4004130}
I0619 00:11:59.850855 25668 heartbeater.cc:346] Connected to a master server at 127.24.119.61:46325
W0619 00:11:59.851076 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 3e7c4b61f5314e40ad71371155a50b05 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3e7c4b61f5314e40ad71371155a50b05 already in progress: creating tablet
I0619 00:11:59.851318 25071 catalog_manager.cc:2891] Scheduling retry of 3e7c4b61f5314e40ad71371155a50b05 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 64 ms (attempt = 1)
I0619 00:11:59.851864 25730 tablet_replica.cc:261] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:11:59.853209 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.853417 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.853602 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.853822 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.854049 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.854261 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.854449 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:11:59.854619 25730 raft_consensus.cc:2042] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.855038 25730 raft_consensus.cc:2069] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.855085 25067 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 1066 ms (attempt = 7)
W0619 00:11:59.855376 25067 catalog_manager.cc:2910] Async tablet task 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef consensus state has no leader
I0619 00:11:59.825759 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 331) took 4647ms. Request Metrics: {"tablet-delete.queue_time_us":4621458}
I0619 00:11:59.858728 25727 tablet_bootstrap.cc:427] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:59.858912 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.859062 25727 ts_tablet_manager.cc:1045] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.211s	user 0.004s	sys 0.005s
I0619 00:11:59.848911 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fc7686daae274e0dba87703fe6924805" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:11:59.860806 25736 log.cc:986] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/30a9d9fda8d34caeb8242c372439321b
I0619 00:11:59.860862 25489 raft_consensus.cc:2784] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:11:59.861147 25489 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 3.052s (vote granted)
I0619 00:11:59.861426 25736 ts_tablet_manager.cc:1403] T 30a9d9fda8d34caeb8242c372439321b P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:11:59.861481 25086 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:11:59.862213 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:11:59.863044 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 365) took 3721ms. Request Metrics: {"tablet-delete.queue_time_us":3513345}
I0619 00:11:59.863718 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 366) took 3721ms. Request Metrics: {"tablet-delete.queue_time_us":3714366}
I0619 00:11:59.864320 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 367) took 3714ms. Request Metrics: {"tablet-delete.queue_time_us":3710528}
I0619 00:11:59.865542 25083 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.866201 25083 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.867374 25084 catalog_manager.cc:3663] Got report from deleted tablet 1cfd335c18c047fba2facc5194807abb (table default.table_2af4c8dc619643729ba189a0dd8cd15f [id=ad26a268092e4cc4a44e6845dc511880]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.867713 25084 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.867986 25084 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.868281 25084 catalog_manager.cc:3663] Got report from deleted tablet 6040e369e0fb445fb993d12dd883039b (table default.table_673676f1e1e94d76afe2c9b06d701801 [id=980403da36b04fad89427ba22b96d098]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.868546 25084 catalog_manager.cc:3663] Got report from deleted tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:11:59.870532 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1cfd335c18c047fba2facc5194807abb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.871312 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.871711 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:59.872076 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:59.872448 25084 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:11:59.873292 25087 catalog_manager.cc:2891] Scheduling retry of 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC with a delay of 54 ms (attempt = 1)
W0619 00:11:59.873679 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 551fd06ba0c74bae9a152c7c2ab989f4 consensus state has no leader
I0619 00:11:59.874222 25489 raft_consensus.cc:2245] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:11:59.875241 26145 raft_consensus.cc:457] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:11:59.875921 26145 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.502s (starting election)
I0619 00:11:59.876210 26145 raft_consensus.cc:479] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.877264 26145 leader_election.cc:243] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:11:59.878643 26145 leader_election.cc:243] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:11:59.860412 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cbd67634f5704165b5bb948501697f0c" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:11:59.881683 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cfd335c18c047fba2facc5194807abb" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
W0619 00:11:59.883227 25419 leader_election.cc:306] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.884975 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 518 ms (attempt = 6)
W0619 00:11:59.885232 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.858193 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 333) took 4669ms. Request Metrics: {"tablet-delete.queue_time_us":4668579}
I0619 00:11:59.887455 25727 raft_consensus.cc:325] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.888269 25727 raft_consensus.cc:351] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.888659 25727 raft_consensus.cc:628] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:59.889722 25727 consensus_queue.cc:227] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.891494 25727 ts_tablet_manager.cc:1073] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.000s
I0619 00:11:59.892980 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 332) took 4714ms. Request Metrics: {"tablet-delete.queue_time_us":4696769}
I0619 00:11:59.893754 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 334) took 4636ms. Request Metrics: {"tablet-delete.queue_time_us":4632955}
I0619 00:11:59.894387 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 337) took 4633ms. Request Metrics: {"tablet-delete.queue_time_us":4631857}
I0619 00:11:59.864943 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 368) took 3715ms. Request Metrics: {"tablet-delete.queue_time_us":3702316}
I0619 00:11:59.894981 25741 tablet_replica.cc:261] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:11:59.895213 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:59.895421 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:59.895643 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:59.895467 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 370) took 3723ms. Request Metrics: {"tablet-delete.queue_time_us":3716914}
I0619 00:11:59.895848 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:59.896167 25736 tablet_replica.cc:261] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:11:59.896276 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:59.896488 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:59.896447 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:11:59.896708 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(54373917567a479d93dc8a3fd006d2fe)
I0619 00:11:59.896898 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:11:59.897125 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:11:59.897352 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:11:59.896888 25741 raft_consensus.cc:2042] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.897593 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:11:59.897760 25741 raft_consensus.cc:2069] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.897889 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:11:59.898136 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:11:59.898334 25736 raft_consensus.cc:2042] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I0619 00:11:59.898627 25736 raft_consensus.cc:2069] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I0619 00:11:59.899125 25741 ts_tablet_manager.cc:1369] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.901501 25736 ts_tablet_manager.cc:1369] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.891872 25727 ts_tablet_manager.cc:1028] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:11:59.902941 25727 tablet_bootstrap.cc:427] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:11:59.906146 25727 tablet_bootstrap.cc:578] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:11:59.907912 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.908932 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.909893 25290 leader_election.cc:379] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:11:59.910176 25290 leader_election.cc:270] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:11:59.910661 25940 raft_consensus.cc:2732] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.799s (election complete)
I0619 00:11:59.910886 25940 raft_consensus.cc:2539] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:11:59.911361 25940 raft_consensus.cc:593] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:11:59.911474 25084 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_4864844cc509452e8a846f7b0fc111cf"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:59.911831 25940 consensus_queue.cc:204] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:11:59.912505 25084 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4864844cc509452e8a846f7b0fc111cf in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.913841 25069 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 1042 ms (attempt = 7)
W0619 00:11:59.914100 25069 catalog_manager.cc:2910] Async tablet task 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
I0619 00:11:59.867079 25598 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.916383 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7c4b61f5314e40ad71371155a50b05 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:11:59.917748 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.918524 25067 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 132 ms (attempt = 4)
W0619 00:11:59.918853 25067 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
W0619 00:11:59.919241 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
W0619 00:11:59.919900 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 775f0b6b309c49278066a5e7cf623af2 because the tablet was not found. No further retry: Not found: Tablet not found: 775f0b6b309c49278066a5e7cf623af2
I0619 00:11:59.931391 25331 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.932217 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.920989 25068 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_94f6d4fdfe6c462cb13a00dbdef20cd3 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 136 ms (attempt = 4)
W0619 00:11:59.933271 25068 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_94f6d4fdfe6c462cb13a00dbdef20cd3 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:11:59.925097 25082 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_94f6d4fdfe6c462cb13a00dbdef20cd3" } new_table_name: "default.table_9d96ad27d1d746f9a220f0524cd3c473" alter_external_catalogs: true
I0619 00:11:59.933898 25068 catalog_manager.cc:2891] Scheduling retry of 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC with a delay of 51 ms (attempt = 2)
W0619 00:11:59.934142 25068 catalog_manager.cc:2910] Async tablet task 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 551fd06ba0c74bae9a152c7c2ab989f4 consensus state has no leader
I0619 00:11:59.924721 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b9cda64c37654d10b31d098b9a5e36f2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:11:59.925552 25078 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_cb065cd115b947e5998bc60c601cd99c"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.935132 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b9cda64c37654d10b31d098b9a5e36f2 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:11:59.935298 25078 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cb065cd115b947e5998bc60c601cd99c in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.937950 25085 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.938299 25730 ts_tablet_manager.cc:1369] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:11:59.938604 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
W0619 00:11:59.939538 25071 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:11:59.940510 25724 tablet_bootstrap.cc:427] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:11:59.940899 25724 ts_tablet_manager.cc:1045] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.132s	user 0.003s	sys 0.001s
W0619 00:11:59.941236 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 7f60ea5c5def47ee846d0186bbc90c18 because the tablet was not found. No further retry: Not found: Tablet not found: 7f60ea5c5def47ee846d0186bbc90c18
W0619 00:11:59.942450 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 0ef73ff0abf44e5aa66cee31180c083c with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0ef73ff0abf44e5aa66cee31180c083c already in progress: creating tablet
I0619 00:11:59.942728 25067 catalog_manager.cc:2891] Scheduling retry of 0ef73ff0abf44e5aa66cee31180c083c Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 64 ms (attempt = 1)
W0619 00:11:59.943186 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a85577b0beba4276b963e5c360544bc7 because the tablet was not found. No further retry: Not found: Tablet not found: a85577b0beba4276b963e5c360544bc7
I0619 00:11:59.943635 25724 raft_consensus.cc:325] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.944299 25724 raft_consensus.cc:351] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.944514 25724 raft_consensus.cc:628] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:11:59.945106 25724 consensus_queue.cc:227] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.948271 25087 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:11:59.948736 25087 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]). Expected version 1 got 0
I0619 00:11:59.947922 25724 ts_tablet_manager.cc:1073] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I0619 00:11:59.949576 25082 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 19 ms (attempt = 1)
W0619 00:11:59.949879 25082 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:11:59.950501 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:11:59.952451 25087 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 36 ms (attempt = 1)
W0619 00:11:59.952749 25087 catalog_manager.cc:3945] Failed to send 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.954979 25085 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]). Expected version 1 got 0
I0619 00:11:59.955319 25085 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:11:59.858613 25725 ts_tablet_manager.cc:1028] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:11:59.956181 25085 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 27 ms (attempt = 1)
I0619 00:11:59.956395 25725 tablet_bootstrap.cc:427] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:11:59.956427 25085 catalog_manager.cc:3945] Failed to send 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.956722 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:11:59.959110 25725 tablet_bootstrap.cc:578] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:11:59.915696 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 1cfd335c18c047fba2facc5194807abb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.961091 25462 tablet_service.cc:800] Processing DeleteTablet for tablet fe93c5dd62d94bdda099b9131482c033 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:11:59.961697 25558 leader_election.cc:306] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:11:59.961949 25558 leader_election.cc:270] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:11:59.963485 25085 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:11:59.963548 26266 raft_consensus.cc:2732] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.980s (election complete)
I0619 00:11:59.963798 26266 raft_consensus.cc:2487] T cbd67634f5704165b5bb948501697f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:11:59.964445 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:11:59.966434 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:11:59.967260 25086 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_d2a13e99b1964be7a97603e011c9cb6b" }
I0619 00:11:59.967849 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.970712 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 1c4c69294cb945938004e5b9a0550e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.970357 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_9a15f182e63744ac950a33cf371b30b4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:11:59.971360 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9a15f182e63744ac950a33cf371b30b4 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:11:59.973088 25071 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 58 ms (attempt = 2)
W0619 00:11:59.973376 25071 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:11:59.973695 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) successfully deleted
W0619 00:11:59.974841 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 0a33940e5347489993ab4679e9aeafde because the tablet was not found. No further retry: Not found: Tablet not found: 0a33940e5347489993ab4679e9aeafde
I0619 00:11:59.975109 25727 tablet_bootstrap.cc:427] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:11:59.975469 25727 ts_tablet_manager.cc:1045] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.073s	user 0.005s	sys 0.000s
W0619 00:11:59.975720 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 075076aaa16e4ea8a596a7a8c52f103e because the tablet was not found. No further retry: Not found: Tablet not found: 075076aaa16e4ea8a596a7a8c52f103e
I0619 00:11:59.976881 25598 tablet_service.cc:759] Processing CreateTablet for tablet a7a3d73c048749b489660d109fa15158 (table=default.table_b7a90cb93e8f4fe79cb29ff3ab818928 [id=f7e13f6492c0456e930628e95fd81358]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.977450 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a7a3d73c048749b489660d109fa15158. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.976936 25727 raft_consensus.cc:325] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.977691 25727 raft_consensus.cc:351] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:11:59.977866 25727 raft_consensus.cc:628] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:11:59.978266 25727 consensus_queue.cc:227] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:11:59.984858 25331 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:11:59.986034 25068 catalog_manager.cc:2891] Scheduling retry of 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC with a delay of 67 ms (attempt = 3)
W0619 00:11:59.986299 25068 catalog_manager.cc:2910] Async tablet task 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 551fd06ba0c74bae9a152c7c2ab989f4 consensus state has no leader
I0619 00:11:59.986469 25598 ts_tablet_manager.cc:1173] T a7a3d73c048749b489660d109fa15158 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:11:59.986963 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_ed122503385b4360827d3299026dc91e" }
I0619 00:11:59.987056 25331 tablet_service.cc:759] Processing CreateTablet for tablet a7a3d73c048749b489660d109fa15158 (table=default.table_b7a90cb93e8f4fe79cb29ff3ab818928 [id=f7e13f6492c0456e930628e95fd81358]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.987653 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a7a3d73c048749b489660d109fa15158. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:11:59.988489 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.988472 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cfd335c18c047fba2facc5194807abb" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:11:59.988951 25623 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.146s (vote granted)
I0619 00:11:59.989181 25623 raft_consensus.cc:2245] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:11:59.990952 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 72 ms (attempt = 2)
W0619 00:11:59.991201 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:11:59.991672 25069 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 1045 ms (attempt = 7)
W0619 00:11:59.991958 25069 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
W0619 00:11:59.992241 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 0a33940e5347489993ab4679e9aeafde because the tablet was not found. No further retry: Not found: Tablet not found: 0a33940e5347489993ab4679e9aeafde
I0619 00:11:59.993041 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 8e3a356933e749b9a4ade476ec30f475 with delete_type TABLET_DATA_DELETED (replaced by tablet beddf1523a854f5b9292c0e4013d7436) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:11:59.994074 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 46 ms (attempt = 2)
W0619 00:11:59.994375 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:11:59.994289 26145 consensus_queue.cc:963] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:11:59.994761 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc because the tablet was not found. No further retry: Not found: Tablet not found: 981d862d1bab4b14bb10e6fc2174e1cc
I0619 00:11:59.995812 25603 tablet_service.cc:759] Processing CreateTablet for tablet bee859b75508414daff59e130e1e860a (table=default.table_ef4ccfddaf014d73b60e77f40fd7944b [id=ff5c9a7aabc0476f81b6d5affed01562]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:11:59.996469 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bee859b75508414daff59e130e1e860a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.001744 25725 tablet_bootstrap.cc:427] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:12:00.002111 25725 ts_tablet_manager.cc:1045] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.046s	user 0.004s	sys 0.003s
I0619 00:12:00.003535 25725 raft_consensus.cc:325] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.005862 25724 ts_tablet_manager.cc:1028] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:00.006247 25724 tablet_bootstrap.cc:427] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:00.007232 25603 ts_tablet_manager.cc:1173] T bee859b75508414daff59e130e1e860a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.008832 25724 tablet_bootstrap.cc:578] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:00.010102 25727 ts_tablet_manager.cc:1073] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.034s	user 0.003s	sys 0.000s
I0619 00:12:00.017791 26302 tablet.cc:1200] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:00.020153 25598 tablet_service.cc:759] Processing CreateTablet for tablet 6aac02942b3042cc93a299d1a34b8d86 (table=default.table_5714a0615d4d4f588547fdcf3951168b [id=a0d0a5186dff469ca7100892e99e1fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.020545 25599 tablet_service.cc:759] Processing CreateTablet for tablet faa6e26ad5b5437ea58edd4b6d19c42a (table=default.table_a6a97f8910184441b129e04f28661796 [id=d965eb79ac1944a8801a7cd57f88ceba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.020150 25600 tablet_service.cc:759] Processing CreateTablet for tablet e5b3be4f05474541b222b0c42884ec80 (table=default.table_d2a13e99b1964be7a97603e011c9cb6b [id=08e4e7f0b7864a91ad75b900b2a89186]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:00.023022 25071 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:12:00.024487 25083 master_service.cc:523] ReplaceTablet: received request to replace tablet 17e165e0995a4c029aa31392b9bbc04a from {username='jenkins'} at 127.24.119.62:54390
I0619 00:12:00.026075 25082 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:12:00.026721 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:00.020787 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6aac02942b3042cc93a299d1a34b8d86. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.029161 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet faa6e26ad5b5437ea58edd4b6d19c42a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.032060 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.032784 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5b3be4f05474541b222b0c42884ec80. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.040438 25086 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet e5b3be4f05474541b222b0c42884ec80
I0619 00:12:00.040748 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5b3be4f05474541b222b0c42884ec80 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:00.041116 25080 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet f2d15bd8d6ce460c8422625ba060c6e0
I0619 00:12:00.041126 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5b3be4f05474541b222b0c42884ec80 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:00.041424 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f2d15bd8d6ce460c8422625ba060c6e0 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24)
I0619 00:12:00.041596 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5b3be4f05474541b222b0c42884ec80 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:00.041826 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f2d15bd8d6ce460c8422625ba060c6e0 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24)
I0619 00:12:00.042194 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f2d15bd8d6ce460c8422625ba060c6e0 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24)
I0619 00:12:00.042418 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 2a0b025496ee4087acadc27bbb6d4983
I0619 00:12:00.042486 25080 catalog_manager.cc:4596] ReplaceTablet: tablet f2d15bd8d6ce460c8422625ba060c6e0 deleted and replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24
I0619 00:12:00.042754 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2a0b025496ee4087acadc27bbb6d4983 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:00.043082 26296 raft_consensus.cc:457] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.043213 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2a0b025496ee4087acadc27bbb6d4983 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:00.043350 26296 raft_consensus.cc:2732] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.863s (starting election)
I0619 00:12:00.043709 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2a0b025496ee4087acadc27bbb6d4983 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:00.043602 26296 raft_consensus.cc:479] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.044667 26296 leader_election.cc:243] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.045096 26296 leader_election.cc:243] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.048899 25599 ts_tablet_manager.cc:1173] T faa6e26ad5b5437ea58edd4b6d19c42a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.049902 25730 ts_tablet_manager.cc:1382] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I0619 00:12:00.050017 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0ef73ff0abf44e5aa66cee31180c083c on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:12:00.050678 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 4134 ms (attempt = 9)
W0619 00:12:00.050948 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.050979 25071 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 90 ms (attempt = 3)
W0619 00:12:00.051301 25071 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:00.051376 25067 catalog_manager.cc:2891] Scheduling retry of 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC with a delay of 548 ms (attempt = 6)
I0619 00:12:00.051601 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 80 ms (attempt = 3)
W0619 00:12:00.051851 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:12:00.054751 25068 catalog_manager.cc:2891] Scheduling retry of 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC with a delay of 154 ms (attempt = 4)
W0619 00:12:00.055039 25068 catalog_manager.cc:2910] Async tablet task 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 551fd06ba0c74bae9a152c7c2ab989f4 consensus state has no leader
I0619 00:12:00.012662 25331 ts_tablet_manager.cc:1173] T a7a3d73c048749b489660d109fa15158 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.012926 25330 tablet_service.cc:759] Processing CreateTablet for tablet bee859b75508414daff59e130e1e860a (table=default.table_ef4ccfddaf014d73b60e77f40fd7944b [id=ff5c9a7aabc0476f81b6d5affed01562]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.057749 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bee859b75508414daff59e130e1e860a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.013778 25336 tablet_service.cc:759] Processing CreateTablet for tablet 6aac02942b3042cc93a299d1a34b8d86 (table=default.table_5714a0615d4d4f588547fdcf3951168b [id=a0d0a5186dff469ca7100892e99e1fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.061030 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6aac02942b3042cc93a299d1a34b8d86. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.063643 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 105 ms (attempt = 3)
W0619 00:12:00.063886 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:12:00.014590 25326 tablet_service.cc:759] Processing CreateTablet for tablet 9eb09151d7b24d25bd156192e0a5cd33 (table=default.table_37bb1c669e584d62b5b5a9f562e3f4d6 [id=394af19f05124f939a83155e7e50a5be]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.065013 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9eb09151d7b24d25bd156192e0a5cd33. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.054504 25601 tablet_service.cc:800] Processing DeleteTablet for tablet 2a0b025496ee4087acadc27bbb6d4983 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.054900 25602 tablet_service.cc:800] Processing DeleteTablet for tablet f2d15bd8d6ce460c8422625ba060c6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.066570 25068 catalog_manager.cc:2891] Scheduling retry of 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC with a delay of 2075 ms (attempt = 8)
W0619 00:12:00.066845 25068 catalog_manager.cc:2910] Async tablet task 38c512bb89034576a27eb55bbda35985 (table default.table_493f801c4cb6466ab9ae71ff0b25c30e [id=a8374f43d2fc46968e8368d1ecf7f839]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c512bb89034576a27eb55bbda35985 consensus state has no leader
I0619 00:12:00.066922 25730 log.cc:986] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/30a9d9fda8d34caeb8242c372439321b
I0619 00:12:00.067600 25730 ts_tablet_manager.cc:1403] T 30a9d9fda8d34caeb8242c372439321b P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:12:00.065740 25724 tablet_bootstrap.cc:427] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:00.009697 25736 ts_tablet_manager.cc:1382] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I0619 00:12:00.053689 26231 raft_consensus.cc:457] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.068027 25724 ts_tablet_manager.cc:1045] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.062s	user 0.005s	sys 0.000s
I0619 00:12:00.015417 25323 tablet_service.cc:759] Processing CreateTablet for tablet e5b3be4f05474541b222b0c42884ec80 (table=default.table_d2a13e99b1964be7a97603e011c9cb6b [id=08e4e7f0b7864a91ad75b900b2a89186]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.054288 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 1cfd335c18c047fba2facc5194807abb with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.068236 25736 log.cc:986] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/17e165e0995a4c029aa31392b9bbc04a
I0619 00:12:00.068243 26231 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.590s (starting election)
I0619 00:12:00.068696 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5b3be4f05474541b222b0c42884ec80. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.068913 25736 ts_tablet_manager.cc:1403] T 17e165e0995a4c029aa31392b9bbc04a P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:12:00.068748 26231 raft_consensus.cc:479] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.069844 26231 leader_election.cc:243] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.016252 25325 tablet_service.cc:759] Processing CreateTablet for tablet faa6e26ad5b5437ea58edd4b6d19c42a (table=default.table_a6a97f8910184441b129e04f28661796 [id=d965eb79ac1944a8801a7cd57f88ceba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.070274 26231 leader_election.cc:243] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.070539 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet faa6e26ad5b5437ea58edd4b6d19c42a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.070888 25069 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 540 ms (attempt = 6)
W0619 00:12:00.071156 25069 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:12:00.069739 25724 raft_consensus.cc:325] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.072113 25724 raft_consensus.cc:351] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.072324 25724 raft_consensus.cc:628] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:00.072793 25068 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 278 ms (attempt = 5)
W0619 00:12:00.073063 25068 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:00.072886 25724 consensus_queue.cc:227] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:12:00.073504 25068 catalog_manager.cc:2910] Async tablet task 18dbdb33a25e487ab87ecb85af31fd55 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 18dbdb33a25e487ab87ecb85af31fd55 consensus state has no leader
I0619 00:12:00.074642 25725 raft_consensus.cc:351] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.076280 25724 ts_tablet_manager.cc:1073] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.001s
I0619 00:12:00.056735 25600 ts_tablet_manager.cc:1173] T e5b3be4f05474541b222b0c42884ec80 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.078090 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_58781b6793a04331a5a69cdbb7cd2cc4" } new_table_name: "default.table_c0119de186df44faa67a07b4cd18cf17" alter_external_catalogs: true
I0619 00:12:00.079656 25725 raft_consensus.cc:628] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:12:00.080289 25353 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "29ee1c11105e4d1e9ac74a247cd2953d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.075458 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 377) took 3849ms. Request Metrics: {"tablet-delete.queue_time_us":3666781}
I0619 00:12:00.083642 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 376) took 3858ms. Request Metrics: {"tablet-delete.queue_time_us":3853358}
I0619 00:12:00.084342 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 378) took 3647ms. Request Metrics: {"tablet-delete.queue_time_us":3642191}
I0619 00:12:00.084947 25736 tablet_replica.cc:261] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:12:00.085202 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:12:00.085445 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:12:00.085707 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:12:00.085947 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:12:00.086205 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(6040e369e0fb445fb993d12dd883039b)
W0619 00:12:00.086328 25419 leader_election.cc:306] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.086414 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 262) took 4243ms. Request Metrics: {"tablet-delete.queue_time_us":4007653}
I0619 00:12:00.086439 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:12:00.087085 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(6040e369e0fb445fb993d12dd883039b)
I0619 00:12:00.087281 25736 raft_consensus.cc:2042] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.087545 25736 raft_consensus.cc:2069] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.088966 25736 ts_tablet_manager.cc:1369] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:00.091024 25290 consensus_peers.cc:434] T 17e165e0995a4c029aa31392b9bbc04a P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Couldn't send request to peer 10bda0b4856f4441b5aed24cdc4764a8 for tablet 17e165e0995a4c029aa31392b9bbc04a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17e165e0995a4c029aa31392b9bbc04a. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:00.091009 25603 tablet_service.cc:759] Processing CreateTablet for tablet 9eb09151d7b24d25bd156192e0a5cd33 (table=default.table_37bb1c669e584d62b5b5a9f562e3f4d6 [id=394af19f05124f939a83155e7e50a5be]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:00.091691 25067 catalog_manager.cc:2910] Async tablet task 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 309a14ee0658485b9ea724d07b8cce20 consensus state has no leader
I0619 00:12:00.091540 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 263) took 4248ms. Request Metrics: {"tablet-delete.queue_time_us":4246859}
I0619 00:12:00.092140 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.092214 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 30a9d9fda8d34caeb8242c372439321b (table default.table_45dd93d67cc94f2c8e9e784347d14167 [id=60a6d49e09f04230abe707ed433d2e9a]) successfully deleted
I0619 00:12:00.092541 25067 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 265 ms (attempt = 5)
W0619 00:12:00.092741 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 0a33940e5347489993ab4679e9aeafde because the tablet was not found. No further retry: Not found: Tablet not found: 0a33940e5347489993ab4679e9aeafde
I0619 00:12:00.094008 25725 consensus_queue.cc:227] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.097268 25598 ts_tablet_manager.cc:1173] T 6aac02942b3042cc93a299d1a34b8d86 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:00.099812 25067 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:12:00.101033 25333 tablet_service.cc:800] Processing DeleteTablet for tablet f2d15bd8d6ce460c8422625ba060c6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.101241 25332 tablet_service.cc:800] Processing DeleteTablet for tablet 2a0b025496ee4087acadc27bbb6d4983 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.101439 25080 catalog_manager.cc:3663] Got report from deleted tablet b34043f14f1c4edd97a1d4689ced5b32 (table default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.101886 25332 tablet_service.cc:800] Processing DeleteTablet for tablet 0ef73ff0abf44e5aa66cee31180c083c with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.102083 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:00.102247 25331 tablet_service.cc:800] Processing DeleteTablet for tablet e5b3be4f05474541b222b0c42884ec80 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.103565 25080 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:12:00.104194 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:00.104980 25724 ts_tablet_manager.cc:1028] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:00.105448 25724 tablet_bootstrap.cc:427] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:00.107715 25724 tablet_bootstrap.cc:578] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:00.098412 25599 tablet_service.cc:800] Processing DeleteTablet for tablet e5b3be4f05474541b222b0c42884ec80 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.102876 26302 tablet.cc:1195] T 9e5e691e6d374cd4bba1f18d8c821723 P 395763cf325d4ce19859830a7ad35c85: Already running schema version 1 got alter request for version 1
I0619 00:12:00.109895 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9eb09151d7b24d25bd156192e0a5cd33. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.112696 25080 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:12:00.112702 25730 tablet_replica.cc:261] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:12:00.113061 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(8e3a356933e749b9a4ade476ec30f475)
I0619 00:12:00.113317 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(8e3a356933e749b9a4ade476ec30f475)
I0619 00:12:00.113350 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:00.113629 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(8e3a356933e749b9a4ade476ec30f475)
I0619 00:12:00.122714 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(8e3a356933e749b9a4ade476ec30f475)
I0619 00:12:00.122282 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.122998 25600 tablet_service.cc:759] Processing CreateTablet for tablet 52f90b430595453fb058c67b41140e21 (table=default.table_d2bfd9daf88c4770a6ce099bcdfb3931 [id=4b9143ad2ae24ede92f69cc096248ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.123117 25080 catalog_manager.cc:3663] Got report from deleted tablet 9e5e691e6d374cd4bba1f18d8c821723 (table default.table_3c040b90a5ff40c180e4679dee37fd1f [id=9249a51147764492bf0fb06b5c500da7]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:12:00.123402 25080 catalog_manager.cc:3663] Got report from deleted tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:12:00.123682 25080 catalog_manager.cc:3663] Got report from deleted tablet b8bc8d4a970843258ab5e9530f425d0f (table default.table_3186954a4b3d40d8be527ce49a270d8b [id=eca38c9105fd4a3a8d9b1e260ebc045c]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.123797 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 52f90b430595453fb058c67b41140e21. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.124686 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:00.125051 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.125403 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:00.126200 25599 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.126897 25331 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.127552 25087 catalog_manager.cc:3663] Got report from deleted tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:12:00.127537 25332 tablet_service.cc:759] Processing CreateTablet for tablet 52f90b430595453fb058c67b41140e21 (table=default.table_d2bfd9daf88c4770a6ce099bcdfb3931 [id=4b9143ad2ae24ede92f69cc096248ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.127895 25087 catalog_manager.cc:3663] Got report from deleted tablet e0070a115e8540cda5f999394deac616 (table default.table_c8d425c090594b149578c26728881c9d [id=85de8220b2024813a8fc84ffdd34106a]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:12:00.128154 25087 catalog_manager.cc:3663] Got report from deleted tablet 93930ffe285d48d080a8ca4793e871d9 (table default.table_3f22e2fb4a9e47aca16e414746eecd63 [id=acb8add3e0fd49fab202731910e3d20e]) (Table deleted at 2018-06-19 00:11:53 PDT): Sending delete request for this tablet
I0619 00:12:00.126677 25598 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.129215 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:00.129572 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.129943 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93930ffe285d48d080a8ca4793e871d9 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:00.130502 25599 tablet_service.cc:759] Processing CreateTablet for tablet 14700cbc3fd7493f95ebc770a056a53e (table=default.table_8dda14f89d4a4854a63884924f69321e [id=9b3bff0c663841279279f50cf445a265]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.131340 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14700cbc3fd7493f95ebc770a056a53e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.131405 25727 ts_tablet_manager.cc:1028] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:12:00.131793 25727 tablet_bootstrap.cc:427] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:12:00.132125 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b50379774e4d4fdfb823ab2dc919db30" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.133266 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.134703 25331 tablet_service.cc:759] Processing CreateTablet for tablet 14700cbc3fd7493f95ebc770a056a53e (table=default.table_8dda14f89d4a4854a63884924f69321e [id=9b3bff0c663841279279f50cf445a265]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.134712 25326 ts_tablet_manager.cc:1173] T 9eb09151d7b24d25bd156192e0a5cd33 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.135185 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 154 ms (attempt = 4)
W0619 00:12:00.135445 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:12:00.135740 25325 ts_tablet_manager.cc:1173] T faa6e26ad5b5437ea58edd4b6d19c42a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.136301 25323 ts_tablet_manager.cc:1173] T e5b3be4f05474541b222b0c42884ec80 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.137207 25330 ts_tablet_manager.cc:1173] T bee859b75508414daff59e130e1e860a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.137823 25336 ts_tablet_manager.cc:1173] T 6aac02942b3042cc93a299d1a34b8d86 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.139586 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b50379774e4d4fdfb823ab2dc919db30" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.140722 25725 ts_tablet_manager.cc:1073] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.138s	user 0.004s	sys 0.000s
I0619 00:12:00.141099 25725 ts_tablet_manager.cc:1028] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
W0619 00:12:00.141095 25553 leader_election.cc:306] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.141553 25725 tablet_bootstrap.cc:427] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:12:00.144023 25725 tablet_bootstrap.cc:578] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:12:00.145161 25071 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 143 ms (attempt = 4)
W0619 00:12:00.145437 25071 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:00.146435 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 6040e369e0fb445fb993d12dd883039b with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.146229 25087 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_0cf296f7569545cd8b6dfcfd6b1360b5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:00.147157 25087 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0cf296f7569545cd8b6dfcfd6b1360b5 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.147409 25423 leader_election.cc:379] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.147477 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) successfully deleted
I0619 00:12:00.147794 25423 leader_election.cc:270] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:12:00.147974 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 756057a7c34a4c61a261deaf849e433a with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 756057a7c34a4c61a261deaf849e433a already in progress: creating tablet
I0619 00:12:00.148190 25071 catalog_manager.cc:2891] Scheduling retry of 756057a7c34a4c61a261deaf849e433a Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 65 ms (attempt = 1)
W0619 00:12:00.148581 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 54373917567a479d93dc8a3fd006d2fe because the tablet was not found. No further retry: Not found: Tablet not found: 54373917567a479d93dc8a3fd006d2fe
I0619 00:12:00.123016 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(8e3a356933e749b9a4ade476ec30f475)
I0619 00:12:00.149168 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(8e3a356933e749b9a4ade476ec30f475)
I0619 00:12:00.149370 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(8e3a356933e749b9a4ade476ec30f475)
I0619 00:12:00.149556 25730 raft_consensus.cc:2042] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.149855 25730 raft_consensus.cc:2069] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.150383 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "29ee1c11105e4d1e9ac74a247cd2953d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:12:00.150849 25623 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.127s (vote granted)
I0619 00:12:00.151053 25623 raft_consensus.cc:2245] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:12:00.151088 25730 ts_tablet_manager.cc:1369] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.151710 25598 tablet_service.cc:800] Processing DeleteTablet for tablet b8bc8d4a970843258ab5e9530f425d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.152396 25332 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 52f90b430595453fb058c67b41140e21. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.152588 25741 ts_tablet_manager.cc:1382] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:00.153013 25741 log.cc:986] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/54373917567a479d93dc8a3fd006d2fe
W0619 00:12:00.153434 25558 leader_election.cc:306] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.153702 25558 leader_election.cc:270] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.154119 26296 raft_consensus.cc:2732] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.969s (election complete)
I0619 00:12:00.154353 26296 raft_consensus.cc:2487] T b50379774e4d4fdfb823ab2dc919db30 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.154853 25727 tablet_bootstrap.cc:578] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:12:00.155977 26307 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.858s (election complete)
I0619 00:12:00.156801 26307 raft_consensus.cc:2539] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:00.157043 26307 raft_consensus.cc:457] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:00.157068 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_b7a90cb93e8f4fe79cb29ff3ab818928" } new_table_name: "default.table_71308ace98a941b8922f99345c2300a2" alter_external_catalogs: true
I0619 00:12:00.157282 26307 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.098s (starting election)
I0619 00:12:00.157729 26307 raft_consensus.cc:2784] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.158949 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14700cbc3fd7493f95ebc770a056a53e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.158577 25078 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_d37acd08be134f80aaa9fca2a259551b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:00.159456 25078 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d37acd08be134f80aaa9fca2a259551b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.162825 26307 raft_consensus.cc:479] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:12:00.163884 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:12:00.165699 25603 ts_tablet_manager.cc:1173] T 9eb09151d7b24d25bd156192e0a5cd33 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.166648 25332 ts_tablet_manager.cc:1173] T 52f90b430595453fb058c67b41140e21 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.170215 25068 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 132 ms (attempt = 4)
W0619 00:12:00.170487 25068 catalog_manager.cc:2910] Async tablet task 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 71510e2112fd4dc9ba08de08648d7ad7 consensus state has no leader
I0619 00:12:00.172680 25085 catalog_manager.cc:2891] Scheduling retry of 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC with a delay of 39 ms (attempt = 1)
I0619 00:12:00.172783 25080 catalog_manager.cc:3663] Got report from deleted tablet b34043f14f1c4edd97a1d4689ced5b32 (table default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
W0619 00:12:00.172976 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 455d8e1ee5be484b8005312d2d886821 consensus state has no leader
I0619 00:12:00.173470 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:00.174401 25067 catalog_manager.cc:2891] Scheduling retry of 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC with a delay of 4104 ms (attempt = 9)
W0619 00:12:00.174736 25067 catalog_manager.cc:2910] Async tablet task 26455f6f1a47489cab7576d428344fd8 (table default.table_b2e120aa5da945a0b123e7ce5642ac94 [id=7f358d20fb074f93b40ef2a5ccd2e667]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 26455f6f1a47489cab7576d428344fd8 consensus state has no leader
I0619 00:12:00.180557 25940 raft_consensus.cc:457] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.180838 25940 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.056s (starting election)
I0619 00:12:00.181087 25940 raft_consensus.cc:479] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.181480 25741 ts_tablet_manager.cc:1403] T 54373917567a479d93dc8a3fd006d2fe P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:12:00.182059 25940 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.182452 25940 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.184072 25332 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:12:00.185307 25290 leader_election.cc:306] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: b34043f14f1c4edd97a1d4689ced5b32
I0619 00:12:00.186350 25940 raft_consensus.cc:457] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.186578 25940 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.030s (starting election)
I0619 00:12:00.186825 25940 raft_consensus.cc:479] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.187773 25940 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.188155 25940 leader_election.cc:243] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.189980 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:12:00.190520 25616 raft_consensus.cc:2175] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:12:00.191135 25727 tablet_bootstrap.cc:427] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:12:00.191490 25727 ts_tablet_manager.cc:1045] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.060s	user 0.004s	sys 0.001s
I0619 00:12:00.192328 25724 tablet_bootstrap.cc:427] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:00.192793 25724 ts_tablet_manager.cc:1045] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.088s	user 0.004s	sys 0.000s
I0619 00:12:00.193177 25290 leader_election.cc:393] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:12:00.193929 25086 catalog_manager.cc:2891] Scheduling retry of a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC with a delay of 48 ms (attempt = 1)
W0619 00:12:00.194211 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7a3d73c048749b489660d109fa15158 consensus state has no leader
I0619 00:12:00.195318 25727 raft_consensus.cc:325] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.195971 25727 raft_consensus.cc:351] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.195623 25724 raft_consensus.cc:325] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.196209 25724 raft_consensus.cc:351] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.196225 25727 raft_consensus.cc:628] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:12:00.196390 25724 raft_consensus.cc:628] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:00.196853 25727 consensus_queue.cc:227] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.197541 25600 ts_tablet_manager.cc:1173] T 52f90b430595453fb058c67b41140e21 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.197422 25724 consensus_queue.cc:227] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.199002 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 1032 ms (attempt = 7)
W0619 00:12:00.199276 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:12:00.181340 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.183080 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.203192 25087 catalog_manager.cc:3663] Got report from deleted tablet 756057a7c34a4c61a261deaf849e433a (table default.table_252d88d7d98b495e80affd5dc6eeb739 [id=f8e9404177f44b2cba84c5146b780133]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:12:00.183157 25423 leader_election.cc:379] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.184937 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b34043f14f1c4edd97a1d4689ced5b32" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.203757 25486 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.954s (vote granted)
I0619 00:12:00.203949 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 756057a7c34a4c61a261deaf849e433a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.203994 25486 raft_consensus.cc:2245] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:12:00.205204 25290 leader_election.cc:379] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.205437 25290 leader_election.cc:270] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:00.206104 25940 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.175s (election complete)
I0619 00:12:00.206331 25940 raft_consensus.cc:2539] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:00.206535 25940 raft_consensus.cc:457] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:00.206756 25940 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.943s (starting election)
I0619 00:12:00.206707 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 1061 ms (attempt = 7)
I0619 00:12:00.206948 25940 raft_consensus.cc:2784] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:12:00.206980 25067 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:12:00.189990 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "71510e2112fd4dc9ba08de08648d7ad7" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.207921 25489 raft_consensus.cc:2175] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:12:00.208907 25423 leader_election.cc:270] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:00.209401 26227 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.570s (election complete)
I0619 00:12:00.209651 26227 raft_consensus.cc:2539] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:00.209944 26227 raft_consensus.cc:593] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:12:00.210117 25290 leader_election.cc:393] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 in current term 1: Already voted for candidate 10bda0b4856f4441b5aed24cdc4764a8 in this term.
I0619 00:12:00.210391 25290 leader_election.cc:270] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.210626 25068 catalog_manager.cc:2891] Scheduling retry of 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC with a delay of 282 ms (attempt = 5)
I0619 00:12:00.210369 26227 consensus_queue.cc:204] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:12:00.210917 25068 catalog_manager.cc:2910] Async tablet task 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 551fd06ba0c74bae9a152c7c2ab989f4 consensus state has no leader
I0619 00:12:00.212515 26271 raft_consensus.cc:2732] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.894s (election complete)
I0619 00:12:00.212769 26271 raft_consensus.cc:2784] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.212775 25087 catalog_manager.cc:3663] Got report from deleted tablet 756057a7c34a4c61a261deaf849e433a (table default.table_252d88d7d98b495e80affd5dc6eeb739 [id=f8e9404177f44b2cba84c5146b780133]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:12:00.213143 25940 raft_consensus.cc:479] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.214093 25724 ts_tablet_manager.cc:1073] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.021s	user 0.003s	sys 0.001s
I0619 00:12:00.214306 25940 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.214481 25724 ts_tablet_manager.cc:1028] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:00.214840 25724 tablet_bootstrap.cc:427] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:00.215586 26296 raft_consensus.cc:457] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.215858 26296 raft_consensus.cc:2732] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.970s (starting election)
I0619 00:12:00.215842 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b34043f14f1c4edd97a1d4689ced5b32" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:12:00.216285 25489 raft_consensus.cc:2784] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.216089 26296 raft_consensus.cc:479] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.216522 25489 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.604s (vote granted)
I0619 00:12:00.217039 26296 leader_election.cc:243] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.217036 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 756057a7c34a4c61a261deaf849e433a on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.217404 26296 leader_election.cc:243] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.217751 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 756057a7c34a4c61a261deaf849e433a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.219442 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 756057a7c34a4c61a261deaf849e433a on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.220863 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 756057a7c34a4c61a261deaf849e433a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.221289 25489 raft_consensus.cc:2245] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:12:00.180097 25331 ts_tablet_manager.cc:1173] T 14700cbc3fd7493f95ebc770a056a53e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.220667 26271 raft_consensus.cc:2487] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.203052 25725 tablet_bootstrap.cc:427] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:12:00.222775 25725 ts_tablet_manager.cc:1045] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.081s	user 0.004s	sys 0.001s
I0619 00:12:00.223223 25290 leader_election.cc:379] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.223444 25290 leader_election.cc:270] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:00.225368 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b8bc8d4a970843258ab5e9530f425d0f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.226315 25078 catalog_manager.cc:3663] Got report from deleted tablet f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.226972 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:12:00.227128 25553 consensus_peers.cc:434] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet e60ecc51c92c461889b360dde1dc5cf2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:00.227773 25553 leader_election.cc:306] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.229138 25725 raft_consensus.cc:325] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.229779 26271 raft_consensus.cc:2732] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.679s (election complete)
I0619 00:12:00.230027 26271 raft_consensus.cc:2539] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:00.230754 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 346) took 4602ms. Request Metrics: {"tablet-delete.queue_time_us":4266575}
I0619 00:12:00.232100 25727 ts_tablet_manager.cc:1073] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.040s	user 0.003s	sys 0.000s
I0619 00:12:00.232573 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet 54373917567a479d93dc8a3fd006d2fe (table default.table_ec99a1cc9ff4419bbcdd5d5faa6a12b3 [id=5492a3d5ab3043a9bd457df3426fb2d1]) successfully deleted
I0619 00:12:00.232159 25087 catalog_manager.cc:3786] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:00.233285 25087 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]). Expected version 1 got 0
I0619 00:12:00.234905 25078 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b2aff78ac49c4833bbfccd2f1593204a"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:00.235388 25086 catalog_manager.cc:3663] Got report from deleted tablet f1b566b2b58f4955aa25ae990a4a8a1a (table default.table_1ccfb0785fb045859278f19e637bbdc7 [id=122dad4ddc134d21b827dca8587ed257]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.236026 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:12:00.238761 25078 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b2aff78ac49c4833bbfccd2f1593204a in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.240228 25332 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.231372 25469 tablet_service.cc:759] Processing CreateTablet for tablet a7a3d73c048749b489660d109fa15158 (table=default.table_b7a90cb93e8f4fe79cb29ff3ab818928 [id=f7e13f6492c0456e930628e95fd81358]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.241164 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a7a3d73c048749b489660d109fa15158. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.243371 25071 catalog_manager.cc:2891] Scheduling retry of 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC with a delay of 62 ms (attempt = 2)
W0619 00:12:00.243706 25071 catalog_manager.cc:2910] Async tablet task 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 455d8e1ee5be484b8005312d2d886821 consensus state has no leader
I0619 00:12:00.243849 25727 ts_tablet_manager.cc:1028] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:12:00.244233 25727 tablet_bootstrap.cc:427] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:12:00.237175 25464 tablet_service.cc:759] Processing CreateTablet for tablet 6aac02942b3042cc93a299d1a34b8d86 (table=default.table_5714a0615d4d4f588547fdcf3951168b [id=a0d0a5186dff469ca7100892e99e1fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.244972 25464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6aac02942b3042cc93a299d1a34b8d86. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.239589 25465 tablet_service.cc:759] Processing CreateTablet for tablet e5b3be4f05474541b222b0c42884ec80 (table=default.table_d2a13e99b1964be7a97603e011c9cb6b [id=08e4e7f0b7864a91ad75b900b2a89186]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.245471 25465 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5b3be4f05474541b222b0c42884ec80. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.238158 25467 tablet_service.cc:759] Processing CreateTablet for tablet 9eb09151d7b24d25bd156192e0a5cd33 (table=default.table_37bb1c669e584d62b5b5a9f562e3f4d6 [id=394af19f05124f939a83155e7e50a5be]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.248996 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9eb09151d7b24d25bd156192e0a5cd33. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.233322 25462 tablet_service.cc:759] Processing CreateTablet for tablet bee859b75508414daff59e130e1e860a (table=default.table_ef4ccfddaf014d73b60e77f40fd7944b [id=ff5c9a7aabc0476f81b6d5affed01562]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.252908 25067 catalog_manager.cc:2891] Scheduling retry of a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC with a delay of 76 ms (attempt = 2)
I0619 00:12:00.253134 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bee859b75508414daff59e130e1e860a. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:00.253180 25067 catalog_manager.cc:2910] Async tablet task a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a7a3d73c048749b489660d109fa15158 consensus state has no leader
I0619 00:12:00.239771 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b8bc8d4a970843258ab5e9530f425d0f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.253983 25331 tablet_service.cc:800] Processing DeleteTablet for tablet f1b566b2b58f4955aa25ae990a4a8a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.248409 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 347) took 4620ms. Request Metrics: {"tablet-delete.queue_time_us":4602566}
I0619 00:12:00.254900 25741 tablet_replica.cc:261] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
W0619 00:12:00.255096 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 8ca3a0771b9e41eeb0d14d886f970159 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 8ca3a0771b9e41eeb0d14d886f970159 already in progress: creating tablet
I0619 00:12:00.255312 25067 catalog_manager.cc:2891] Scheduling retry of 8ca3a0771b9e41eeb0d14d886f970159 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 55 ms (attempt = 1)
I0619 00:12:00.256629 25465 ts_tablet_manager.cc:1173] T e5b3be4f05474541b222b0c42884ec80 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.259529 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_f02c65a4cf4e4707a03b3e4a3c9f8c99"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:00.260457 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f02c65a4cf4e4707a03b3e4a3c9f8c99 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.240602 26307 leader_election.cc:243] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.262431 26296 raft_consensus.cc:457] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.262729 26296 raft_consensus.cc:2732] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.221s (starting election)
I0619 00:12:00.262974 26296 raft_consensus.cc:479] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.263253 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41e052fdd92b4c3d9803fa43433e60d5" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:12:00.255138 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:12:00.264286 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:12:00.264503 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:12:00.264725 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(d6c3c7f783cb43d59b46502261aeced3)
W0619 00:12:00.264658 25419 leader_election.cc:306] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.264727 25080 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_0cf296f7569545cd8b6dfcfd6b1360b5" }
I0619 00:12:00.264957 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:12:00.265019 26307 leader_election.cc:243] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.265506 25940 leader_election.cc:243] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.266631 25467 ts_tablet_manager.cc:1173] T 9eb09151d7b24d25bd156192e0a5cd33 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.268594 25727 tablet_bootstrap.cc:578] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:12:00.269435 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_f7cff01960d64d739ae50955fabe7254" }
I0619 00:12:00.270381 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:12:00.270612 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(d6c3c7f783cb43d59b46502261aeced3)
I0619 00:12:00.270918 25462 ts_tablet_manager.cc:1173] T bee859b75508414daff59e130e1e860a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.271478 25725 raft_consensus.cc:351] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.271736 25725 raft_consensus.cc:628] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:12:00.272225 25725 consensus_queue.cc:227] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.274273 25741 raft_consensus.cc:2042] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.274828 25068 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC with a delay of 4097 ms (attempt = 9)
W0619 00:12:00.275105 25068 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:12:00.275487 25741 raft_consensus.cc:2069] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.273722 25736 ts_tablet_manager.cc:1382] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:00.276935 25463 tablet_service.cc:759] Processing CreateTablet for tablet faa6e26ad5b5437ea58edd4b6d19c42a (table=default.table_a6a97f8910184441b129e04f28661796 [id=d965eb79ac1944a8801a7cd57f88ceba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.280282 25736 log.cc:986] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/6040e369e0fb445fb993d12dd883039b
I0619 00:12:00.280782 25736 ts_tablet_manager.cc:1403] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:12:00.282327 26296 leader_election.cc:243] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.278486 25464 ts_tablet_manager.cc:1173] T 6aac02942b3042cc93a299d1a34b8d86 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.282806 26296 leader_election.cc:243] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:12:00.283640 25290 leader_election.cc:306] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: b34043f14f1c4edd97a1d4689ced5b32
I0619 00:12:00.283865 25599 ts_tablet_manager.cc:1173] T 14700cbc3fd7493f95ebc770a056a53e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.284137 25462 tablet_service.cc:800] Processing DeleteTablet for tablet e5b3be4f05474541b222b0c42884ec80 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:12:00.285208 25558 leader_election.cc:306] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.285470 25558 leader_election.cc:270] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.285938 26296 raft_consensus.cc:2732] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.187s (election complete)
I0619 00:12:00.286180 26296 raft_consensus.cc:2487] T b8bc8d4a970843258ab5e9530f425d0f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.287225 25469 ts_tablet_manager.cc:1173] T a7a3d73c048749b489660d109fa15158 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.290215 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet bae6d308dae54b0394a364d2988ec6b3
I0619 00:12:00.290415 25080 catalog_manager.cc:4020] Not sending DeleteTablet requests; no consensus state for tablet 1e0cb53e599c41eab8eae819608ac894
I0619 00:12:00.290504 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bae6d308dae54b0394a364d2988ec6b3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.290613 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet faa6e26ad5b5437ea58edd4b6d19c42a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.290925 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bae6d308dae54b0394a364d2988ec6b3 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.291303 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bae6d308dae54b0394a364d2988ec6b3 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.291373 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 278 ms (attempt = 5)
W0619 00:12:00.291651 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:12:00.291960 25071 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 298 ms (attempt = 5)
W0619 00:12:00.292834 25071 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:00.296026 25086 catalog_manager.cc:3663] Got report from deleted tablet e9a221a1d9814ef99a0b4ccaa04d1017 (table default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1): Sending delete request for this tablet
I0619 00:12:00.296725 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
I0619 00:12:00.282150 26266 raft_consensus.cc:457] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.297581 26266 raft_consensus.cc:2732] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.544s (starting election)
I0619 00:12:00.297986 25599 tablet_service.cc:800] Processing DeleteTablet for tablet bae6d308dae54b0394a364d2988ec6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.297863 26266 raft_consensus.cc:479] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.298883 26266 leader_election.cc:243] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.298961 25331 tablet_service.cc:800] Processing DeleteTablet for tablet bae6d308dae54b0394a364d2988ec6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.299288 26266 leader_election.cc:243] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.303387 25086 catalog_manager.cc:3663] Got report from deleted tablet e9a221a1d9814ef99a0b4ccaa04d1017 (table default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1): Sending delete request for this tablet
I0619 00:12:00.303951 25727 tablet_bootstrap.cc:427] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:12:00.304085 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9a221a1d9814ef99a0b4ccaa04d1017 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1)
W0619 00:12:00.304302 25069 catalog_manager.cc:2910] Async tablet task 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 0a7470bfeab041f9895e491231b57eb7 consensus state has no leader
I0619 00:12:00.304381 25727 ts_tablet_manager.cc:1045] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.060s	user 0.005s	sys 0.001s
I0619 00:12:00.305217 25741 ts_tablet_manager.cc:1369] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.306198 25724 tablet_bootstrap.cc:578] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:00.306228 25463 ts_tablet_manager.cc:1173] T faa6e26ad5b5437ea58edd4b6d19c42a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.307194 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 379) took 3865ms. Request Metrics: {"tablet-delete.queue_time_us":3634329}
I0619 00:12:00.308221 26271 raft_consensus.cc:593] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
I0619 00:12:00.308739 26271 consensus_queue.cc:204] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.311189 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 383) took 3697ms. Request Metrics: {"tablet-delete.queue_time_us":3681455}
I0619 00:12:00.311403 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:12:00.311971 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 385) took 3611ms. Request Metrics: {"tablet-delete.queue_time_us":3597422}
I0619 00:12:00.312666 25736 tablet_replica.cc:261] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:12:00.311530 26266 raft_consensus.cc:457] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.312930 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:00.313009 26266 raft_consensus.cc:2732] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.902s (starting election)
I0619 00:12:00.313271 26266 raft_consensus.cc:479] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.314237 26266 leader_election.cc:243] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.314635 26266 leader_election.cc:243] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.315237 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.314546 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41e052fdd92b4c3d9803fa43433e60d5" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:12:00.315527 25616 raft_consensus.cc:2784] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.315786 25616 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.613s (vote granted)
I0619 00:12:00.309303 25727 raft_consensus.cc:325] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.316496 25727 raft_consensus.cc:351] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.316669 25727 raft_consensus.cc:628] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:12:00.317152 25727 consensus_queue.cc:227] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.318624 25727 ts_tablet_manager.cc:1073] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I0619 00:12:00.318739 25730 ts_tablet_manager.cc:1382] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:00.318925 25727 ts_tablet_manager.cc:1028] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:12:00.318989 25730 log.cc:986] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/8e3a356933e749b9a4ade476ec30f475
I0619 00:12:00.319258 25727 tablet_bootstrap.cc:427] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:12:00.319756 25071 catalog_manager.cc:2891] Scheduling retry of 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC with a delay of 106 ms (attempt = 3)
W0619 00:12:00.320055 25071 catalog_manager.cc:2910] Async tablet task 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 455d8e1ee5be484b8005312d2d886821 consensus state has no leader
I0619 00:12:00.320137 25616 raft_consensus.cc:2245] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:12:00.323065 25724 tablet_bootstrap.cc:427] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:00.324034 25724 ts_tablet_manager.cc:1045] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.109s	user 0.005s	sys 0.000s
I0619 00:12:00.324710 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:00.325330 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:00.322293 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "31b36efa3dbd4aa983b3a396aa9d62f1" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.326144 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:00.319442 25730 ts_tablet_manager.cc:1403] T 8e3a356933e749b9a4ade476ec30f475 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:12:00.326706 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:00.327076 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 6040e369e0fb445fb993d12dd883039b (table default.table_673676f1e1e94d76afe2c9b06d701801 [id=980403da36b04fad89427ba22b96d098]) successfully deleted
I0619 00:12:00.324256 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 9e5e691e6d374cd4bba1f18d8c821723 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:58 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.286849 25725 ts_tablet_manager.cc:1073] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.064s	user 0.004s	sys 0.000s
W0619 00:12:00.327525 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet eb36a22cf80841b58fd5ce9e6998bf00 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet eb36a22cf80841b58fd5ce9e6998bf00 already in progress: creating tablet
I0619 00:12:00.327776 25725 ts_tablet_manager.cc:1028] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:12:00.327680 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 260) took 4485ms. Request Metrics: {"tablet-delete.queue_time_us":4252118}
I0619 00:12:00.327805 25071 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 60 ms (attempt = 1)
I0619 00:12:00.328160 25725 tablet_bootstrap.cc:427] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
W0619 00:12:00.328382 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 0c1f35568fe24c6c8cd2b88871b4520d with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0c1f35568fe24c6c8cd2b88871b4520d already in progress: creating tablet
I0619 00:12:00.328634 25071 catalog_manager.cc:2891] Scheduling retry of 0c1f35568fe24c6c8cd2b88871b4520d Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 22 ms (attempt = 1)
I0619 00:12:00.330458 25067 catalog_manager.cc:2891] Scheduling retry of a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC with a delay of 94 ms (attempt = 3)
W0619 00:12:00.330731 25067 catalog_manager.cc:2910] Async tablet task a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a7a3d73c048749b489660d109fa15158 consensus state has no leader
I0619 00:12:00.330538 25725 tablet_bootstrap.cc:578] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:12:00.288373 25462 tablet_service.cc:800] Processing DeleteTablet for tablet f2d15bd8d6ce460c8422625ba060c6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.323946 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 2a0b025496ee4087acadc27bbb6d4983 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.2:51757
W0619 00:12:00.323606 25423 leader_election.cc:306] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 6040e369e0fb445fb993d12dd883039b
I0619 00:12:00.332805 25423 leader_election.cc:270] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:12:00.333313 25423 raft_consensus.cc:2442] T 6040e369e0fb445fb993d12dd883039b P 10bda0b4856f4441b5aed24cdc4764a8: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:12:00.334235 25730 tablet_replica.cc:261] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:12:00.327260 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:00.335659 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:00.335860 25736 raft_consensus.cc:2042] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:00.336066 25736 consensus_peers.cc:456] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:12:00.325722 25724 raft_consensus.cc:325] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.336271 25736 consensus_peers.cc:456] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:12:00.336674 25736 raft_consensus.cc:2069] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.336519 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(93930ffe285d48d080a8ca4793e871d9)
I0619 00:12:00.337210 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(93930ffe285d48d080a8ca4793e871d9)
I0619 00:12:00.337702 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(93930ffe285d48d080a8ca4793e871d9)
I0619 00:12:00.338140 25736 ts_tablet_manager.cc:1369] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.336743 25724 raft_consensus.cc:351] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.338387 25724 raft_consensus.cc:628] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:00.339022 25724 consensus_queue.cc:227] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.340447 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 8e3a356933e749b9a4ade476ec30f475 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) successfully deleted
I0619 00:12:00.344867 25489 raft_consensus.cc:1081] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer 395763cf325d4ce19859830a7ad35c85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:00.347824 25086 catalog_manager.cc:3663] Got report from deleted tablet b34043f14f1c4edd97a1d4689ced5b32 (table default.table_eb0f587538da43c39a0017ca0930cda9 [id=d4faaea1fb384667952cdd6532f78217]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.348459 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:00.350224 25331 tablet_service.cc:800] Processing DeleteTablet for tablet b34043f14f1c4edd97a1d4689ced5b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.351501 25069 catalog_manager.cc:2891] Scheduling retry of 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC with a delay of 2070 ms (attempt = 8)
W0619 00:12:00.351788 25069 catalog_manager.cc:2910] Async tablet task 13265ab1253f44d6b3ffe61fa0141be1 (table default.table_d8b2d7e633094119b0fa40e3972594f8 [id=6d61f318c5e74320b6c45eeecc382d5b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 13265ab1253f44d6b3ffe61fa0141be1 consensus state has no leader
I0619 00:12:00.349349 25463 tablet_service.cc:759] Processing CreateTablet for tablet 52f90b430595453fb058c67b41140e21 (table=default.table_d2bfd9daf88c4770a6ce099bcdfb3931 [id=4b9143ad2ae24ede92f69cc096248ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.338191 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(93930ffe285d48d080a8ca4793e871d9)
I0619 00:12:00.352365 25068 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 555 ms (attempt = 6)
I0619 00:12:00.352421 25086 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
W0619 00:12:00.352664 25068 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:00.352900 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(93930ffe285d48d080a8ca4793e871d9)
I0619 00:12:00.353173 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:12:00.353322 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(93930ffe285d48d080a8ca4793e871d9)
I0619 00:12:00.353976 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0c1f35568fe24c6c8cd2b88871b4520d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 8140397d408e47129a182fb6255a0a39)
I0619 00:12:00.354686 25331 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.354566 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(93930ffe285d48d080a8ca4793e871d9)
I0619 00:12:00.355090 25730 raft_consensus.cc:2042] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.355346 25730 raft_consensus.cc:2069] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.356503 25730 ts_tablet_manager.cc:1369] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.359541 25086 catalog_manager.cc:3663] Got report from deleted tablet d0624684b6b04ba3be260b2705dc00e1 (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded): Sending delete request for this tablet
I0619 00:12:00.360260 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:12:00.360251 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "31b36efa3dbd4aa983b3a396aa9d62f1" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.360976 25069 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 1073 ms (attempt = 7)
W0619 00:12:00.361239 25069 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:12:00.361922 25599 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:12:00.364531 25558 leader_election.cc:306] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.364637 25086 catalog_manager.cc:3663] Got report from deleted tablet d0624684b6b04ba3be260b2705dc00e1 (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded): Sending delete request for this tablet
I0619 00:12:00.365321 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:12:00.367054 25599 tablet_service.cc:800] Processing DeleteTablet for tablet d0624684b6b04ba3be260b2705dc00e1 with delete_type TABLET_DATA_DELETED (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.367089 25724 ts_tablet_manager.cc:1073] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.043s	user 0.003s	sys 0.001s
I0619 00:12:00.367475 25724 ts_tablet_manager.cc:1028] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:00.368410 25462 tablet_service.cc:759] Processing CreateTablet for tablet 14700cbc3fd7493f95ebc770a056a53e (table=default.table_8dda14f89d4a4854a63884924f69321e [id=9b3bff0c663841279279f50cf445a265]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.339777 26313 raft_consensus.cc:457] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.369191 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 52f90b430595453fb058c67b41140e21. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.369191 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f155a418d0d547f7b775360f1a90ce01" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.370836 25067 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 560 ms (attempt = 6)
W0619 00:12:00.371095 25067 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:12:00.372046 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14700cbc3fd7493f95ebc770a056a53e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.383146 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ef1566aa89b24620af70f2322df0dd56" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.387845 25423 leader_election.cc:379] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.388128 25423 leader_election.cc:270] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:00.389430 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet eb36a22cf80841b58fd5ce9e6998bf00 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.393527 25727 tablet_bootstrap.cc:578] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:12:00.397753 25331 tablet_service.cc:759] Processing CreateTablet for tablet 07029a4b8b7a46aa8656d6bbeaee975a (table=default.table_cb065cd115b947e5998bc60c601cd99c [id=551e48e9d60440dc9de25ed881037c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.377698 26313 raft_consensus.cc:2732] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.868s (starting election)
I0619 00:12:00.398344 25331 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 07029a4b8b7a46aa8656d6bbeaee975a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.400128 25741 ts_tablet_manager.cc:1382] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:00.400727 25741 log.cc:986] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/d6c3c7f783cb43d59b46502261aeced3
W0619 00:12:00.401124 25290 consensus_peers.cc:434] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet fc7686daae274e0dba87703fe6924805. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc7686daae274e0dba87703fe6924805. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:00.401229 25741 ts_tablet_manager.cc:1403] T d6c3c7f783cb43d59b46502261aeced3 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:12:00.367789 26266 consensus_queue.cc:963] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:00.370410 25553 leader_election.cc:306] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.402745 25553 leader_election.cc:270] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:12:00.403934 25553 leader_election.cc:306] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:12:00.404552 25553 leader_election.cc:306] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.398310 25599 tablet_service.cc:759] Processing CreateTablet for tablet 07029a4b8b7a46aa8656d6bbeaee975a (table=default.table_cb065cd115b947e5998bc60c601cd99c [id=551e48e9d60440dc9de25ed881037c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.406158 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 07029a4b8b7a46aa8656d6bbeaee975a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.408161 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_89031abc2b4847ae883f7fb4ae304e9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:00.408289 25078 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_b9eef2bdee184eeca723789e224e92ec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:00.408639 26307 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.010s (election complete)
I0619 00:12:00.408890 26307 raft_consensus.cc:2539] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
W0619 00:12:00.409097 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_89031abc2b4847ae883f7fb4ae304e9b in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.409088 26307 raft_consensus.cc:457] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:00.409276 26307 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.544s (starting election)
I0619 00:12:00.409464 26307 raft_consensus.cc:2784] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.409639 25087 master_service.cc:523] ReplaceTablet: received request to replace tablet d62715e230154e2abe126ef97feec7ce from {username='jenkins'} at 127.24.119.62:54390
I0619 00:12:00.409814 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_54f94ed54aa24b75bf10f8f57d8d1621" } new_table_name: "default.table_0b76fb6c648d42f191b24676254f8b8c" alter_external_catalogs: true
I0619 00:12:00.410517 26296 raft_consensus.cc:2732] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.881s (election complete)
I0619 00:12:00.410789 26296 raft_consensus.cc:2487] T 31b36efa3dbd4aa983b3a396aa9d62f1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.411458 25085 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_f18ea472f11c4efa81d1e00b99a5a22a" } new_table_name: "default.table_12960b64943d44eda1316d619b74b8eb" alter_external_catalogs: true
W0619 00:12:00.409098 25078 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b9eef2bdee184eeca723789e224e92ec in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.415514 26307 raft_consensus.cc:479] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.417369 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f155a418d0d547f7b775360f1a90ce01" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.418467 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 1dd3f7afe05f49199b18220f9c11ccd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.420192 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e0070a115e8540cda5f999394deac616 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.416504 26307 leader_election.cc:243] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.421917 26307 leader_election.cc:243] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.422741 25725 tablet_bootstrap.cc:427] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:12:00.422858 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cfd335c18c047fba2facc5194807abb" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
W0619 00:12:00.424208 25419 leader_election.cc:306] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.425148 25462 ts_tablet_manager.cc:1173] T 14700cbc3fd7493f95ebc770a056a53e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.426993 25067 catalog_manager.cc:2891] Scheduling retry of a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC with a delay of 164 ms (attempt = 4)
W0619 00:12:00.427279 25067 catalog_manager.cc:2910] Async tablet task a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a7a3d73c048749b489660d109fa15158 consensus state has no leader
I0619 00:12:00.428321 25724 tablet_bootstrap.cc:427] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:00.428730 25463 ts_tablet_manager.cc:1173] T 52f90b430595453fb058c67b41140e21 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.430950 25085 catalog_manager.cc:2891] Scheduling retry of a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC with a delay of 44 ms (attempt = 1)
I0619 00:12:00.431093 25071 catalog_manager.cc:2891] Scheduling retry of 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC with a delay of 154 ms (attempt = 4)
W0619 00:12:00.431233 25085 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4c3d22a3237408387518e3061470d01 consensus state has no leader
W0619 00:12:00.431382 25071 catalog_manager.cc:2910] Async tablet task 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 455d8e1ee5be484b8005312d2d886821 consensus state has no leader
I0619 00:12:00.433091 25724 tablet_bootstrap.cc:578] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:00.435412 25599 ts_tablet_manager.cc:1173] T 07029a4b8b7a46aa8656d6bbeaee975a P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.436625 25736 ts_tablet_manager.cc:1382] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:00.436883 25736 log.cc:986] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/3e7724f967ae41d7b523055d1a4f58db
W0619 00:12:00.439009 25558 leader_election.cc:306] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.439275 25558 leader_election.cc:270] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.439751 26266 raft_consensus.cc:2732] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.575s (election complete)
I0619 00:12:00.439977 26266 raft_consensus.cc:2487] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.445302 26314 raft_consensus.cc:457] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.446259 25067 catalog_manager.cc:2891] Scheduling retry of ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC with a delay of 4133 ms (attempt = 9)
W0619 00:12:00.446529 25067 catalog_manager.cc:2910] Async tablet task ec8f89d6855544a584eb2ebcd88b7fb6 (table default.table_462e0451fdd448aab866e586f244c58f [id=e81ccf2cf02a4d278717d5fd8274f2d6]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet ec8f89d6855544a584eb2ebcd88b7fb6 consensus state has no leader
I0619 00:12:00.449064 25085 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:12:00.449792 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:12:00.453817 25085 catalog_manager.cc:3663] Got report from deleted tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:12:00.454445 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.454978 25332 tablet_service.cc:759] Processing CreateTablet for tablet 687c893ccc6346debb555de59d3e5341 (table=default.table_defc57494cc44555be460e4b77d34fe8 [id=f7a57d952d8a4cdf8c7ecdf22e85df45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.455577 25332 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 687c893ccc6346debb555de59d3e5341. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.456529 25330 tablet_service.cc:800] Processing DeleteTablet for tablet 17e165e0995a4c029aa31392b9bbc04a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.456495 25336 tablet_service.cc:759] Processing CreateTablet for tablet 24281f894abb44419f18834dca2de65e (table=default.table_4eb928c7853a4940ba9a83ba7f30a181 [id=388ad92a190b457497ffa5304c077739]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.457193 25336 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 24281f894abb44419f18834dca2de65e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.458195 25730 ts_tablet_manager.cc:1382] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:00.401875 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 349) took 4773ms. Request Metrics: {"tablet-delete.queue_time_us":4625032}
I0619 00:12:00.459595 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet d6c3c7f783cb43d59b46502261aeced3 (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) successfully deleted
I0619 00:12:00.462796 25330 tablet_service.cc:759] Processing CreateTablet for tablet 81f562d9bb3c4213b3970e0d79416bf0 (table=default.table_18b6ad6c2b1142c4a307bb3a4d56f328 [id=e878f682a0404fa5a518e9e0d120d456]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.463321 25330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 81f562d9bb3c4213b3970e0d79416bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.445539 26314 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.676s (starting election)
I0619 00:12:00.419106 26313 raft_consensus.cc:479] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.467038 25323 tablet_service.cc:759] Processing CreateTablet for tablet 740cd8ff815e4fd18466b0ca7fcb4b24 (table=default.table_45234f5ba11946f390d40924c779fe3e [id=12a7e07939ae4125b30d7473b37e1695]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.467608 25323 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 740cd8ff815e4fd18466b0ca7fcb4b24. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.468138 25325 tablet_service.cc:759] Processing CreateTablet for tablet b4f30bbe544f4f1f84ace8aab6adb319 (table=default.table_9a15f182e63744ac950a33cf371b30b4 [id=19547e0cea314ab8b0231bd548caf15e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.468695 25325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b4f30bbe544f4f1f84ace8aab6adb319. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.463847 26314 raft_consensus.cc:479] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.469503 26314 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.469907 26314 leader_election.cc:243] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.476014 25068 catalog_manager.cc:2891] Scheduling retry of a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC with a delay of 34 ms (attempt = 2)
W0619 00:12:00.476287 25068 catalog_manager.cc:2910] Async tablet task a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a4c3d22a3237408387518e3061470d01 consensus state has no leader
I0619 00:12:00.479856 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_ed223177099d4877b476a5ec777d1d71"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:00.480769 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ed223177099d4877b476a5ec777d1d71 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.483057 25351 raft_consensus.cc:1081] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:12:00.461298 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cfd335c18c047fba2facc5194807abb" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:12:00.472538 25603 tablet_service.cc:759] Processing CreateTablet for tablet 81f562d9bb3c4213b3970e0d79416bf0 (table=default.table_18b6ad6c2b1142c4a307bb3a4d56f328 [id=e878f682a0404fa5a518e9e0d120d456]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.484745 25616 raft_consensus.cc:2784] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.485003 25616 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.032s (vote granted)
I0619 00:12:00.474099 25598 tablet_service.cc:759] Processing CreateTablet for tablet 740cd8ff815e4fd18466b0ca7fcb4b24 (table=default.table_45234f5ba11946f390d40924c779fe3e [id=12a7e07939ae4125b30d7473b37e1695]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.474642 25602 tablet_service.cc:759] Processing CreateTablet for tablet b4f30bbe544f4f1f84ace8aab6adb319 (table=default.table_9a15f182e63744ac950a33cf371b30b4 [id=19547e0cea314ab8b0231bd548caf15e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.473017 25600 tablet_service.cc:759] Processing CreateTablet for tablet 24281f894abb44419f18834dca2de65e (table=default.table_4eb928c7853a4940ba9a83ba7f30a181 [id=388ad92a190b457497ffa5304c077739]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.475194 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:12:00.488889 25623 raft_consensus.cc:2206] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.490188 25290 leader_election.cc:393] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344). Message: Invalid argument: T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.485102 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 81f562d9bb3c4213b3970e0d79416bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.492105 25616 raft_consensus.cc:2245] T 1cfd335c18c047fba2facc5194807abb P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:12:00.492611 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b4f30bbe544f4f1f84ace8aab6adb319. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.493347 25598 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 740cd8ff815e4fd18466b0ca7fcb4b24. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.495468 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_0b76fb6c648d42f191b24676254f8b8c" }
I0619 00:12:00.495848 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 24281f894abb44419f18834dca2de65e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.499869 25068 catalog_manager.cc:2891] Scheduling retry of 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC with a delay of 538 ms (attempt = 6)
W0619 00:12:00.500149 25068 catalog_manager.cc:2910] Async tablet task 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 551fd06ba0c74bae9a152c7c2ab989f4 consensus state has no leader
I0619 00:12:00.500367 25078 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_b9eef2bdee184eeca723789e224e92ec" }
I0619 00:12:00.502688 25730 log.cc:986] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/93930ffe285d48d080a8ca4793e871d9
I0619 00:12:00.503408 25730 ts_tablet_manager.cc:1403] T 93930ffe285d48d080a8ca4793e871d9 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:12:00.503981 25598 ts_tablet_manager.cc:1173] T 740cd8ff815e4fd18466b0ca7fcb4b24 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.504525 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_38a9cc3b684f4e64a71452318b1dfdb1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:00.505414 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_38a9cc3b684f4e64a71452318b1dfdb1 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:12:00.505893 25067 catalog_manager.cc:2910] Async tablet task 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c4c69294cb945938004e5b9a0550e72 consensus state has no leader
I0619 00:12:00.423127 25725 ts_tablet_manager.cc:1045] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.095s	user 0.004s	sys 0.001s
I0619 00:12:00.437250 25489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ef1566aa89b24620af70f2322df0dd56" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.438264 25486 raft_consensus.cc:1081] T fc7686daae274e0dba87703fe6924805 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:00.477571 25599 tablet_service.cc:759] Processing CreateTablet for tablet 687c893ccc6346debb555de59d3e5341 (table=default.table_defc57494cc44555be460e4b77d34fe8 [id=f7a57d952d8a4cdf8c7ecdf22e85df45]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.507983 26314 consensus_queue.cc:963] T fc7686daae274e0dba87703fe6924805 P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:00.508215 25724 tablet_bootstrap.cc:427] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:00.508594 25724 ts_tablet_manager.cc:1045] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.080s	user 0.004s	sys 0.001s
I0619 00:12:00.508227 25725 raft_consensus.cc:325] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.508874 25725 raft_consensus.cc:351] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.509064 25725 raft_consensus.cc:628] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:12:00.478209 25464 tablet_service.cc:800] Processing DeleteTablet for tablet bae6d308dae54b0394a364d2988ec6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.509773 25725 consensus_queue.cc:227] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.493018 25467 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.511008 25068 catalog_manager.cc:2891] Scheduling retry of a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC with a delay of 102 ms (attempt = 3)
W0619 00:12:00.511270 25068 catalog_manager.cc:2910] Async tablet task a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a4c3d22a3237408387518e3061470d01 consensus state has no leader
I0619 00:12:00.511888 25725 ts_tablet_manager.cc:1073] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:12:00.471770 25488 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e60ecc51c92c461889b360dde1dc5cf2" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.512240 25725 ts_tablet_manager.cc:1028] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:12:00.512292 25488 raft_consensus.cc:2206] T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.512629 25725 tablet_bootstrap.cc:427] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:12:00.513481 25290 leader_election.cc:393] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512). Message: Invalid argument: T e60ecc51c92c461889b360dde1dc5cf2 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c007a00e65ad41a79da1a0c1050c55b4 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.483214 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e9a221a1d9814ef99a0b4ccaa04d1017 with delete_type TABLET_DATA_DELETED (replaced by tablet 31b36efa3dbd4aa983b3a396aa9d62f1) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.513763 25290 leader_election.cc:270] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.452189 25462 tablet_service.cc:800] Processing DeleteTablet for tablet 756057a7c34a4c61a261deaf849e433a with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.458776 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 93930ffe285d48d080a8ca4793e871d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:53 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.502377 25423 leader_election.cc:379] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.514277 25423 leader_election.cc:270] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:00.514468 26314 raft_consensus.cc:2732] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.559s (election complete)
I0619 00:12:00.514708 26314 raft_consensus.cc:2784] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.514748 26227 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.943s (election complete)
I0619 00:12:00.514982 26227 raft_consensus.cc:2539] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:00.515194 26227 raft_consensus.cc:457] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:00.515399 26227 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.900s (starting election)
I0619 00:12:00.515740 26227 raft_consensus.cc:2784] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.517745 25736 ts_tablet_manager.cc:1403] T 3e7724f967ae41d7b523055d1a4f58db P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:12:00.521256 25080 catalog_manager.cc:3663] Got report from deleted tablet f155a418d0d547f7b775360f1a90ce01 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55): Sending delete request for this tablet
I0619 00:12:00.521550 25080 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.521980 25078 catalog_manager.cc:4020] Not sending DeleteTablet requests; no consensus state for tablet 7c5ec3a614eb409d8dfe3d5ade823db8
I0619 00:12:00.522406 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f155a418d0d547f7b775360f1a90ce01 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55)
I0619 00:12:00.522790 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:00.523195 25326 tablet_service.cc:759] Processing CreateTablet for tablet 553a468dfae9489c8c11b784b55adef6 (table=default.table_4864844cc509452e8a846f7b0fc111cf [id=de883858812b45c493afaef046e18865]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.523856 25326 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 553a468dfae9489c8c11b784b55adef6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.505363 25603 ts_tablet_manager.cc:1173] T 81f562d9bb3c4213b3970e0d79416bf0 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.524009 26227 raft_consensus.cc:479] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.525025 26227 leader_election.cc:243] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.525846 26227 leader_election.cc:243] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.526561 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 7277051d52dd4be1a6dfa06488c98293
I0619 00:12:00.526891 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7277051d52dd4be1a6dfa06488c98293 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.527276 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7277051d52dd4be1a6dfa06488c98293 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.527806 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7277051d52dd4be1a6dfa06488c98293 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.527637 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "29ee1c11105e4d1e9ac74a247cd2953d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:12:00.528059 25333 tablet_service.cc:800] Processing DeleteTablet for tablet 7277051d52dd4be1a6dfa06488c98293 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.528136 25080 catalog_manager.cc:3663] Got report from deleted tablet f155a418d0d547f7b775360f1a90ce01 (table default.table_6486a9f255374893bfb9f02f620c80da [id=65652a104e3943e69b26a960c2bb1e62]) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55): Sending delete request for this tablet
I0619 00:12:00.528806 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f155a418d0d547f7b775360f1a90ce01 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55)
I0619 00:12:00.529600 25725 tablet_bootstrap.cc:578] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
W0619 00:12:00.530468 25419 leader_election.cc:306] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.532308 25603 tablet_service.cc:800] Processing DeleteTablet for tablet 7277051d52dd4be1a6dfa06488c98293 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.542002 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 384) took 3842ms. Request Metrics: {"tablet-delete.queue_time_us":3596633}
W0619 00:12:00.543190 25558 leader_election.cc:306] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.543473 25558 leader_election.cc:270] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.540937 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 265) took 4669ms. Request Metrics: {"tablet-delete.queue_time_us":4462176}
I0619 00:12:00.544386 25086 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:12:00.545081 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:12:00.545040 25603 tablet_service.cc:759] Processing CreateTablet for tablet 553a468dfae9489c8c11b784b55adef6 (table=default.table_4864844cc509452e8a846f7b0fc111cf [id=de883858812b45c493afaef046e18865]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.545776 26223 raft_consensus.cc:2732] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.851s (election complete)
I0619 00:12:00.548224 26223 raft_consensus.cc:2487] T ef1566aa89b24620af70f2322df0dd56 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.547732 25600 ts_tablet_manager.cc:1173] T 24281f894abb44419f18834dca2de65e P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.548718 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 93930ffe285d48d080a8ca4793e871d9 (table default.table_3f22e2fb4a9e47aca16e414746eecd63 [id=acb8add3e0fd49fab202731910e3d20e]) successfully deleted
I0619 00:12:00.549288 25730 tablet_replica.cc:261] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:12:00.549535 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(e9a221a1d9814ef99a0b4ccaa04d1017)
I0619 00:12:00.553555 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(e9a221a1d9814ef99a0b4ccaa04d1017)
I0619 00:12:00.553771 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(e9a221a1d9814ef99a0b4ccaa04d1017)
I0619 00:12:00.553956 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(e9a221a1d9814ef99a0b4ccaa04d1017)
I0619 00:12:00.554168 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(e9a221a1d9814ef99a0b4ccaa04d1017)
I0619 00:12:00.554374 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(e9a221a1d9814ef99a0b4ccaa04d1017)
I0619 00:12:00.554569 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(e9a221a1d9814ef99a0b4ccaa04d1017)
I0619 00:12:00.554772 25730 raft_consensus.cc:2042] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.555023 25730 raft_consensus.cc:2069] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.556229 25730 ts_tablet_manager.cc:1369] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.557770 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 394) took 3592ms. Request Metrics: {"tablet-delete.queue_time_us":3591263}
I0619 00:12:00.470193 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 345) took 4842ms. Request Metrics: {"tablet-delete.queue_time_us":4839999}
I0619 00:12:00.562945 25741 tablet_replica.cc:261] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:12:00.563210 25600 tablet_service.cc:759] Processing CreateTablet for tablet f2f581e6ef854e14800111a598987abd (table=default.table_b9cda64c37654d10b31d098b9a5e36f2 [id=3b25beee28d4405da5eced08eb9431ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.564074 25333 tablet_service.cc:759] Processing CreateTablet for tablet f2f581e6ef854e14800111a598987abd (table=default.table_b9cda64c37654d10b31d098b9a5e36f2 [id=3b25beee28d4405da5eced08eb9431ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.564749 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f2f581e6ef854e14800111a598987abd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.571849 25333 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f2f581e6ef854e14800111a598987abd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.572969 25086 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.573139 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 520 ms (attempt = 6)
W0619 00:12:00.573487 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:12:00.574026 25736 tablet_replica.cc:261] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:12:00.574427 25080 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:12:00.552259 26307 consensus_queue.cc:963] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:00.575006 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:00.569008 25463 tablet_service.cc:800] Processing DeleteTablet for tablet 0c1f35568fe24c6c8cd2b88871b4520d with delete_type TABLET_DATA_DELETED (replaced by tablet 8140397d408e47129a182fb6255a0a39) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.575137 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:12:00.545346 26227 raft_consensus.cc:457] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.576063 26227 raft_consensus.cc:2732] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.731s (starting election)
I0619 00:12:00.576357 26227 raft_consensus.cc:479] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.577407 26227 leader_election.cc:243] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.578918 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "93930ffe285d48d080a8ca4793e871d9" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.579602 26227 leader_election.cc:243] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.579808 25336 ts_tablet_manager.cc:1173] T 24281f894abb44419f18834dca2de65e P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:00.580914 26317 log.cc:667] Time spent T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
W0619 00:12:00.581488 25419 leader_election.cc:306] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.581974 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:12:00.582247 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:12:00.582470 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:12:00.582716 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(d3c7114381c0423982b9ccbff120c2a2)
W0619 00:12:00.582921 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 0a33940e5347489993ab4679e9aeafde because the tablet was not found. No further retry: Not found: Tablet not found: 0a33940e5347489993ab4679e9aeafde
I0619 00:12:00.582998 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:12:00.583286 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:12:00.583528 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(d3c7114381c0423982b9ccbff120c2a2)
I0619 00:12:00.583758 25741 raft_consensus.cc:2042] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.584053 25741 raft_consensus.cc:2069] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.575775 26313 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.585686 25325 ts_tablet_manager.cc:1173] T b4f30bbe544f4f1f84ace8aab6adb319 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.587023 25333 ts_tablet_manager.cc:1173] T f2f581e6ef854e14800111a598987abd P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.587275 25334 tablet_service.cc:759] Processing CreateTablet for tablet 558473fa04c4429289371ce2ae643421 (table=default.table_32abea053929403b9fd5b7a0b6189479 [id=479771692d41462cad49460cd7051e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.587951 25334 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 558473fa04c4429289371ce2ae643421. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.588564 25080 catalog_manager.cc:3663] Got report from deleted tablet fc7686daae274e0dba87703fe6924805 (table default.table_1074470b12de48baa6a1264ad788d237 [id=7bf3a411e3384c2e83d8c19e21acef4d]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:00.588601 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:00.589205 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:00.550772 25725 tablet_bootstrap.cc:427] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:12:00.591771 25071 catalog_manager.cc:2891] Scheduling retry of 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC with a delay of 305 ms (attempt = 5)
W0619 00:12:00.592068 25071 catalog_manager.cc:2910] Async tablet task 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 455d8e1ee5be484b8005312d2d886821 consensus state has no leader
I0619 00:12:00.592114 26313 leader_election.cc:243] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.592795 25071 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 520 ms (attempt = 6)
I0619 00:12:00.593173 26314 raft_consensus.cc:2487] T e60ecc51c92c461889b360dde1dc5cf2 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.593328 25602 ts_tablet_manager.cc:1173] T b4f30bbe544f4f1f84ace8aab6adb319 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.594033 25599 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 687c893ccc6346debb555de59d3e5341. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.596923 25616 raft_consensus.cc:1081] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:12:00.599958 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 553a468dfae9489c8c11b784b55adef6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.600505 25330 ts_tablet_manager.cc:1173] T 81f562d9bb3c4213b3970e0d79416bf0 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.601207 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:00.602969 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "29ee1c11105e4d1e9ac74a247cd2953d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85"
I0619 00:12:00.603557 25616 raft_consensus.cc:2784] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.603842 25616 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.780s (vote granted)
W0619 00:12:00.606874 25290 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
I0619 00:12:00.607331 25333 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.607744 25616 raft_consensus.cc:2245] T 29ee1c11105e4d1e9ac74a247cd2953d P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 1.
I0619 00:12:00.607858 25741 ts_tablet_manager.cc:1369] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.608129 25323 ts_tablet_manager.cc:1173] T 740cd8ff815e4fd18466b0ca7fcb4b24 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:00.612082 26316 log.cc:667] Time spent T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Append to log took a long time: real 0.274s	user 0.019s	sys 0.001s
I0619 00:12:00.612277 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:00.612514 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:00.612797 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:00.605674 25725 ts_tablet_manager.cc:1045] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.093s	user 0.004s	sys 0.002s
I0619 00:12:00.547097 25724 raft_consensus.cc:325] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.614493 25724 raft_consensus.cc:351] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.594383 25067 catalog_manager.cc:2891] Scheduling retry of a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC with a delay of 274 ms (attempt = 5)
I0619 00:12:00.611799 25069 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 1055 ms (attempt = 7)
I0619 00:12:00.613858 25068 catalog_manager.cc:2891] Scheduling retry of a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC with a delay of 174 ms (attempt = 4)
W0619 00:12:00.614812 25067 catalog_manager.cc:2910] Async tablet task a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a7a3d73c048749b489660d109fa15158 consensus state has no leader
I0619 00:12:00.614701 25724 raft_consensus.cc:628] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
W0619 00:12:00.614830 25069 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
W0619 00:12:00.615602 25071 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:00.615511 25724 consensus_queue.cc:227] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.616763 25602 tablet_service.cc:759] Processing CreateTablet for tablet 558473fa04c4429289371ce2ae643421 (table=default.table_32abea053929403b9fd5b7a0b6189479 [id=479771692d41462cad49460cd7051e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.616806 25080 catalog_manager.cc:3663] Got report from deleted tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) (Table deleted at 2018-06-19 00:11:56 PDT): Sending delete request for this tablet
I0619 00:12:00.617462 25080 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:00.617964 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 558473fa04c4429289371ce2ae643421. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:00.617822 25290 leader_election.cc:306] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
W0619 00:12:00.614864 25068 catalog_manager.cc:2910] Async tablet task a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a4c3d22a3237408387518e3061470d01 consensus state has no leader
I0619 00:12:00.620153 25724 ts_tablet_manager.cc:1073] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.105s	user 0.004s	sys 0.000s
I0619 00:12:00.618521 25290 leader_election.cc:270] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.620440 25080 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]). Expected version 1 got 0
I0619 00:12:00.621191 25080 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 44 ms (attempt = 1)
W0619 00:12:00.621486 25080 catalog_manager.cc:3945] Failed to send 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.622239 26314 raft_consensus.cc:2732] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.924s (election complete)
I0619 00:12:00.622468 26314 raft_consensus.cc:2487] T edfdedbd4eca4b62be24b1649f820d0e P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.623090 25724 ts_tablet_manager.cc:1028] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:00.623415 25724 tablet_bootstrap.cc:427] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:00.621752 25725 raft_consensus.cc:325] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.624572 25725 raft_consensus.cc:351] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.624795 25725 raft_consensus.cc:628] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:12:00.625239 25725 consensus_queue.cc:227] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.626956 25724 tablet_bootstrap.cc:578] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:00.628688 25086 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]). Expected version 1 got 0
I0619 00:12:00.629348 25086 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 40 ms (attempt = 1)
W0619 00:12:00.629602 25086 catalog_manager.cc:3945] Failed to send 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.631791 25080 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_e04e900269934d1581d998ebe954a450"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:00.633832 25080 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e04e900269934d1581d998ebe954a450 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.634866 25332 ts_tablet_manager.cc:1173] T 687c893ccc6346debb555de59d3e5341 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.636554 25326 ts_tablet_manager.cc:1173] T 553a468dfae9489c8c11b784b55adef6 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.637213 25086 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_4eb928c7853a4940ba9a83ba7f30a181" } new_table_name: "default.table_7ead71299b534c358768a664635bb74d" alter_external_catalogs: true
I0619 00:12:00.638849 25068 catalog_manager.cc:2891] Scheduling retry of 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC with a delay of 2062 ms (attempt = 8)
W0619 00:12:00.639158 25068 catalog_manager.cc:2910] Async tablet task 82f1b4e87c3249a6af8a1f27f10a639e (table default.table_54592b380c5f49acac05077eb5114c27 [id=2bfabb7d0cf44dc5a112e214dbbc21b4]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f1b4e87c3249a6af8a1f27f10a639e consensus state has no leader
I0619 00:12:00.641073 25067 catalog_manager.cc:2891] Scheduling retry of 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC with a delay of 1038 ms (attempt = 7)
I0619 00:12:00.640862 25085 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_272eb92753c343cdaca8e3d0f3f117b6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:00.641386 25067 catalog_manager.cc:2910] Async tablet task 309a14ee0658485b9ea724d07b8cce20 (table default.table_c7eb124f548947d7940b1de0352abd23 [id=1e349cc1896d47d6832c30fc440e54aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 309a14ee0658485b9ea724d07b8cce20 consensus state has no leader
W0619 00:12:00.641785 25085 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_272eb92753c343cdaca8e3d0f3f117b6 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.647929 25331 ts_tablet_manager.cc:1173] T 07029a4b8b7a46aa8656d6bbeaee975a P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.648501 25087 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_947e41e56412414da7590c66fe5fc49a" } new_table_name: "default.table_4814ad0a42d246bda1bbfe321580b01f" alter_external_catalogs: true
I0619 00:12:00.653825 25727 tablet_bootstrap.cc:427] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:12:00.654193 25727 ts_tablet_manager.cc:1045] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.335s	user 0.008s	sys 0.011s
I0619 00:12:00.655113 25082 master_service.cc:523] ReplaceTablet: received request to replace tablet f155a418d0d547f7b775360f1a90ce01 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:12:00.658452 25334 ts_tablet_manager.cc:1173] T 558473fa04c4429289371ce2ae643421 P c007a00e65ad41a79da1a0c1050c55b4: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.660068 25725 ts_tablet_manager.cc:1073] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.047s	user 0.004s	sys 0.000s
I0619 00:12:00.663763 26245 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.535s (election complete)
I0619 00:12:00.664186 26245 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:00.663542 25727 raft_consensus.cc:325] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.664808 25727 raft_consensus.cc:351] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.665148 25727 raft_consensus.cc:628] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:12:00.666335 26264 raft_consensus.cc:457] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb25d1bba5bf4c64914f300f122ec898)
I0619 00:12:00.666604 26264 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.744s (starting election)
I0619 00:12:00.666891 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 44 ms (attempt = 2)
W0619 00:12:00.667340 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.627245 25598 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.644784 26223 raft_consensus.cc:457] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.665935 26242 sys_catalog.cc:337] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb25d1bba5bf4c64914f300f122ec898. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:12:00.668140 26223 raft_consensus.cc:2732] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.242s (starting election)
I0619 00:12:00.668292 26242 sys_catalog.cc:340] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:00.666873 26264 raft_consensus.cc:479] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } }
I0619 00:12:00.668406 26223 raft_consensus.cc:479] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.669378 26223 leader_election.cc:243] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.669396 26264 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587)
I0619 00:12:00.669800 26223 leader_election.cc:243] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.669823 26264 leader_election.cc:243] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058)
I0619 00:12:00.671152 25069 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 71 ms (attempt = 2)
I0619 00:12:00.671144 25225 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 308 } } ignore_live_leader: false dest_uuid: "8c39479445934adb9bdbb6e60c3bf712" is_pre_election: true
W0619 00:12:00.671414 25069 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.671715 25225 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.672636 26023 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" instance_seqno: 1529392307113761) as {username='jenkins'} at 127.24.119.2:48241; Asking this server to re-register.
I0619 00:12:00.633388 25599 ts_tablet_manager.cc:1173] T 687c893ccc6346debb555de59d3e5341 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:00.673122 26023 rpcz_store.cc:251] Call kudu.master.MasterService.TSHeartbeat from 127.24.119.2:48241 (request call id 20) took 2079ms (client timeout 1000).
I0619 00:12:00.672658 25890 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c39479445934adb9bdbb6e60c3bf712 (127.24.119.60:57058). Message: Invalid argument: T 00000000000000000000000000000000 P 8c39479445934adb9bdbb6e60c3bf712 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.674330 25603 ts_tablet_manager.cc:1173] T 553a468dfae9489c8c11b784b55adef6 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.671712 25353 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "756057a7c34a4c61a261deaf849e433a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.680707 25725 ts_tablet_manager.cc:1028] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:12:00.681310 25725 tablet_bootstrap.cc:427] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:12:00.613025 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:00.681710 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(9d44681acacf4f04b6fcfbd4c9a18e87)
W0619 00:12:00.681593 25553 leader_election.cc:306] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.681903 25736 raft_consensus.cc:2042] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:00.682032 25463 tablet_service.cc:800] Processing DeleteTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.682186 25736 consensus_peers.cc:456] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:12:00.683871 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 3e7724f967ae41d7b523055d1a4f58db (table default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]) successfully deleted
W0619 00:12:00.684478 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 17e165e0995a4c029aa31392b9bbc04a because the tablet was not found. No further retry: Not found: Tablet not found: 17e165e0995a4c029aa31392b9bbc04a
W0619 00:12:00.685418 26023 rpcz_store.cc:255] Trace:
0619 00:11:58.593354 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:11:58.607269 (+ 13915us) service_pool.cc:221] Handling call
0619 00:12:00.670898 (+2063629us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b4c000b021c. stack: 00007f4d0f86d614 00007f4d0f86d510 00007f4d0f2471b4 00007f4d17f43074 00007f4d17f43009 00007f4d1306f038 00007f4d17f19c7f 00007f4d17f7848c 00007f4d16840a84 00007f4d16840a11 00007f4d16840924 00007f4d110ecce9 00007f4d110ec632 00007f4d110edb8c 00007f4d110efa69 00007f4d110ef9ba
0619 00:12:00.671719 (+   821us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b4c00000f20. stack: 00007f4d0f86d614 00007f4d0f86d510 00007f4d0f2471b4 00007f4d17f43074 00007f4d17f43009 00007f4d17f19e9c 00007f4d17f7848c 00007f4d16840a84 00007f4d16840a11 00007f4d16840924 00007f4d110ecce9 00007f4d110ec632 00007f4d110edb8c 00007f4d110efa69 00007f4d110ef9ba 00007f4d110ef943
0619 00:12:00.673070 (+  1351us) inbound_call.cc:162] Queueing success response
Metrics: {"spinlock_wait_cycles":4922547328}
I0619 00:12:00.682760 25727 consensus_queue.cc:227] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.688078 25736 consensus_peers.cc:456] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:12:00.691155 25736 raft_consensus.cc:2069] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.693652 25600 ts_tablet_manager.cc:1173] T f2f581e6ef854e14800111a598987abd P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.689944 26314 raft_consensus.cc:457] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.693986 26314 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.080s (starting election)
I0619 00:12:00.694231 26314 raft_consensus.cc:479] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.695195 26314 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.695593 25741 ts_tablet_manager.cc:1382] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:00.696465 25083 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_cb73054071fc44cf8f90838b548659d2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:00.696982 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f1b566b2b58f4955aa25ae990a4a8a1a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
W0619 00:12:00.697393 25083 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cb73054071fc44cf8f90838b548659d2 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.697456 25486 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.084s (vote granted)
I0619 00:12:00.697702 25486 raft_consensus.cc:2245] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 0.
I0619 00:12:00.700101 26227 raft_consensus.cc:457] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.700479 26227 raft_consensus.cc:2732] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.153s (starting election)
I0619 00:12:00.700776 26227 raft_consensus.cc:479] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.701220 25602 ts_tablet_manager.cc:1173] T 558473fa04c4429289371ce2ae643421 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:00.702087 26227 leader_election.cc:243] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.702600 25736 ts_tablet_manager.cc:1369] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.704260 25097 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2a016cc9f274bc1979068306ec6d15e" candidate_term: 2 candidate_status { last_received { term: 1 index: 308 } } ignore_live_leader: false dest_uuid: "cb25d1bba5bf4c64914f300f122ec898" is_pre_election: true
I0619 00:12:00.704869 25097 raft_consensus.cc:2206] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.705940 25078 catalog_manager.cc:3663] Got report from deleted tablet cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6): Sending delete request for this tablet
I0619 00:12:00.707237 25078 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb3c60fa2d8f4cafa12a5b0bc806433f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6)
I0619 00:12:00.703990 26227 leader_election.cc:243] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.706568 25725 tablet_bootstrap.cc:578] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:12:00.708204 25892 leader_election.cc:393] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Vote denied by peer cb25d1bba5bf4c64914f300f122ec898 (127.24.119.62:37587). Message: Invalid argument: T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b2a016cc9f274bc1979068306ec6d15e for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:00.671300 26016 rpcz_store.cc:265] Call kudu.master.MasterService.ListTables from 127.24.119.61:54209 (request call id 41) took 2067ms. Request Metrics: {"spinlock_wait_cycles":4936675200}
I0619 00:12:00.712098 25892 leader_election.cc:270] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.712612 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 70 ms (attempt = 3)
W0619 00:12:00.711189 26316 log.cc:667] Time spent T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I0619 00:12:00.713006 26264 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e: Snoozing failure detection for 0.850s (election complete)
W0619 00:12:00.712913 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.713642 26264 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b2a016cc9f274bc1979068306ec6d15e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:00.717304 25082 catalog_manager.cc:3663] Got report from deleted tablet cb3c60fa2d8f4cafa12a5b0bc806433f (table default.table_047cd2779baa44c18f0d11f4e4ae83a2 [id=2b7639db7e3441539055b4caee80db07]) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6): Sending delete request for this tablet
I0619 00:12:00.717635 25082 catalog_manager.cc:3663] Got report from deleted tablet 0a7470bfeab041f9895e491231b57eb7 (table default.table_824bd974d1ae4fc398285c3a2393333f [id=1e6786c5c70b44d6ba8ad6a83b670b1e]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:12:00.718585 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb3c60fa2d8f4cafa12a5b0bc806433f on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6)
I0619 00:12:00.719270 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "756057a7c34a4c61a261deaf849e433a" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.719002 25082 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:00.720616 25463 tablet_service.cc:759] Processing CreateTablet for tablet 07029a4b8b7a46aa8656d6bbeaee975a (table=default.table_cb065cd115b947e5998bc60c601cd99c [id=551e48e9d60440dc9de25ed881037c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.723166 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 07029a4b8b7a46aa8656d6bbeaee975a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.726440 25087 catalog_manager.cc:2891] Scheduling retry of 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC with a delay of 20 ms (attempt = 1)
W0619 00:12:00.727330 25087 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 894de151fa3b4a4fbc962a5cafbccd3a consensus state has no leader
I0619 00:12:00.732556 25085 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_f05cbe4e96c04214a2f1a82f57ee9b77" }
I0619 00:12:00.736671 25725 tablet_bootstrap.cc:427] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:12:00.737800 25086 catalog_manager.cc:2891] Scheduling retry of 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC with a delay of 31 ms (attempt = 1)
W0619 00:12:00.738103 25086 catalog_manager.cc:3998] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24281f894abb44419f18834dca2de65e consensus state has no leader
I0619 00:12:00.738867 25087 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:54390:
table { table_name: "default.table_d036ca047e034c39958950ab500d6004" }
I0619 00:12:00.740900 25725 ts_tablet_manager.cc:1045] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.060s	user 0.005s	sys 0.001s
I0619 00:12:00.701400 26223 raft_consensus.cc:457] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.741745 26223 raft_consensus.cc:2732] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.247s (starting election)
I0619 00:12:00.742007 26223 raft_consensus.cc:479] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.742141 25725 raft_consensus.cc:325] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.742763 25725 raft_consensus.cc:351] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.742939 25725 raft_consensus.cc:628] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:12:00.742995 26223 leader_election.cc:243] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.743378 26223 leader_election.cc:243] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.713204 26296 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.743988 26296 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.814s (starting election)
I0619 00:12:00.744326 25069 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 71 ms (attempt = 3)
W0619 00:12:00.744621 25069 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.744235 26296 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.745182 26296 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.745551 26296 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.749363 25086 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:54390:
name: "default.table_9932f993230644f484e23f0394689f84"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:00.750128 25463 ts_tablet_manager.cc:1173] T 07029a4b8b7a46aa8656d6bbeaee975a P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:00.750313 25086 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9932f993230644f484e23f0394689f84 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:00.695570 26314 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.751278 25741 log.cc:986] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-0/wal/wals/d3c7114381c0423982b9ccbff120c2a2
I0619 00:12:00.699303 25290 leader_election.cc:379] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.703855 25353 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9a221a1d9814ef99a0b4ccaa04d1017" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.746642 25352 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.751972 25741 ts_tablet_manager.cc:1403] T d3c7114381c0423982b9ccbff120c2a2 P c007a00e65ad41a79da1a0c1050c55b4: Deleting consensus metadata
I0619 00:12:00.737898 25423 leader_election.cc:379] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.752022 25290 leader_election.cc:270] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:00.752290 25423 leader_election.cc:270] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
W0619 00:12:00.752488 25290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 53212.5us
I0619 00:12:00.752522 26314 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.930s (election complete)
I0619 00:12:00.752847 26314 raft_consensus.cc:2539] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:00.746284 25354 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d0624684b6b04ba3be260b2705dc00e1" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.753783 26227 raft_consensus.cc:2732] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.900s (election complete)
I0619 00:12:00.754016 26227 raft_consensus.cc:2539] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:00.754324 26227 raft_consensus.cc:593] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:12:00.753046 26314 raft_consensus.cc:457] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:00.754535 26314 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.770s (starting election)
I0619 00:12:00.754609 25352 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.905s (vote granted)
I0619 00:12:00.754776 26314 raft_consensus.cc:2784] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.755007 25352 raft_consensus.cc:2245] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 1.
I0619 00:12:00.754747 26227 consensus_queue.cc:204] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.755306 25083 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]). Expected version 1 got 0
W0619 00:12:00.756917 25553 leader_election.cc:306] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:12:00.757398 25290 leader_election.cc:306] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: f1b566b2b58f4955aa25ae990a4a8a1a
I0619 00:12:00.757527 25553 leader_election.cc:379] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 pre-election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.757766 25553 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won.
I0619 00:12:00.758201 26296 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.769s (election complete)
I0619 00:12:00.758419 26296 raft_consensus.cc:2539] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I0619 00:12:00.758616 26296 raft_consensus.cc:457] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:00.758833 26296 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.786s (starting election)
I0619 00:12:00.759027 26296 raft_consensus.cc:2784] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Advancing to term 2
W0619 00:12:00.759402 25419 leader_election.cc:306] T e9a221a1d9814ef99a0b4ccaa04d1017 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.763588 26296 raft_consensus.cc:479] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.764626 26296 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.765508 26314 raft_consensus.cc:479] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.766464 26314 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.766579 25352 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4"
I0619 00:12:00.767164 25352 raft_consensus.cc:2784] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Advancing to term 2
I0619 00:12:00.767833 25352 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.361s (vote granted)
I0619 00:12:00.770190 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f1b566b2b58f4955aa25ae990a4a8a1a" candidate_uuid: "c007a00e65ad41a79da1a0c1050c55b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:12:00.770375 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 348) took 5142ms. Request Metrics: {"mutex_wait_us":18758,"tablet-delete.queue_time_us":4932696}
I0619 00:12:00.770602 25486 raft_consensus.cc:2784] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:00.770865 25486 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.370s (vote granted)
I0619 00:12:00.771160 25069 catalog_manager.cc:2891] Scheduling retry of 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC with a delay of 42 ms (attempt = 2)
W0619 00:12:00.771445 25069 catalog_manager.cc:2910] Async tablet task 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 24281f894abb44419f18834dca2de65e consensus state has no leader
I0619 00:12:00.772053 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 350) took 5143ms. Request Metrics: {"tablet-delete.queue_time_us":5141054}
I0619 00:12:00.772771 25085 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet ef1566aa89b24620af70f2322df0dd56
I0619 00:12:00.773074 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ef1566aa89b24620af70f2322df0dd56 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.773469 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ef1566aa89b24620af70f2322df0dd56 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.773936 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ef1566aa89b24620af70f2322df0dd56 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.772040 25352 raft_consensus.cc:2245] T 1c4c69294cb945938004e5b9a0550e72 P c007a00e65ad41a79da1a0c1050c55b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 2.
I0619 00:12:00.700170 25727 ts_tablet_manager.cc:1073] T 41e052fdd92b4c3d9803fa43433e60d5 P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.046s	user 0.006s	sys 0.000s
I0619 00:12:00.775116 25727 ts_tablet_manager.cc:1028] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:12:00.775487 25727 tablet_bootstrap.cc:427] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:12:00.775413 25486 raft_consensus.cc:2245] T f1b566b2b58f4955aa25ae990a4a8a1a P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c007a00e65ad41a79da1a0c1050c55b4 in term 1.
I0619 00:12:00.775549 25553 leader_election.cc:379] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 election: Vote granted by peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.775797 25553 leader_election.cc:270] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
I0619 00:12:00.776298 26223 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.821s (election complete)
I0619 00:12:00.776515 26223 raft_consensus.cc:2539] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 2 FOLLOWER]: Leader election won for term 2
I0619 00:12:00.776811 26223 raft_consensus.cc:593] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [term 2 LEADER]: Becoming Leader. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Running, Role: LEADER
I0619 00:12:00.766872 26314 leader_election.cc:243] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Requesting vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.777216 26223 consensus_queue.cc:204] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.778241 25290 leader_election.cc:379] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.778450 25290 leader_election.cc:270] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:00.778910 26314 raft_consensus.cc:2732] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 1.816s (election complete)
I0619 00:12:00.779213 26314 raft_consensus.cc:2539] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:00.779671 26314 raft_consensus.cc:593] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [term 1 LEADER]: Becoming Leader. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Running, Role: LEADER
W0619 00:12:00.780308 25558 leader_election.cc:306] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.780550 25558 leader_election.cc:270] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.781319 26266 raft_consensus.cc:2732] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.938s (election complete)
I0619 00:12:00.781560 26266 raft_consensus.cc:2487] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.781764 25067 catalog_manager.cc:3091] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): tablet d3c7114381c0423982b9ccbff120c2a2 (table default.table_3828fbda3c26454f85595d8e6fbb70e5 [id=be6e839e6cc046409703930d84c28bb8]) successfully deleted
W0619 00:12:00.782368 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb because the tablet was not found. No further retry: Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
I0619 00:12:00.731798 26014 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "395763cf325d4ce19859830a7ad35c85" instance_seqno: 1529392308087416) as {username='jenkins'} at 127.24.119.3:37305; Asking this server to re-register.
W0619 00:12:00.783711 26014 rpcz_store.cc:251] Call kudu.master.MasterService.TSHeartbeat from 127.24.119.3:37305 (request call id 22) took 1927ms (client timeout 1000).
I0619 00:12:00.783845 26314 consensus_queue.cc:204] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.713045 26009 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" instance_seqno: 1529392307113761) as {username='jenkins'} at 127.24.119.2:48241; Asking this server to re-register.
W0619 00:12:00.785683 26009 rpcz_store.cc:251] Call kudu.master.MasterService.TSHeartbeat from 127.24.119.2:48241 (request call id 22) took 838ms (client timeout 1000).
I0619 00:12:00.774504 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 351) took 5139ms. Request Metrics: {"tablet-delete.queue_time_us":5138897}
I0619 00:12:00.789355 26296 leader_election.cc:243] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.743355 25725 consensus_queue.cc:227] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.791975 25725 ts_tablet_manager.cc:1073] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.051s	user 0.003s	sys 0.001s
W0619 00:12:00.792702 26014 rpcz_store.cc:255] Trace:
0619 00:11:58.856185 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:11:58.856320 (+   135us) service_pool.cc:221] Handling call
0619 00:12:00.712851 (+1856531us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b4c00000f20. stack: 00007f4d0f86d614 00007f4d0f86d510 00007f4d0f2471b4 00007f4d17f43074 00007f4d17f43009 00007f4d17f19e9c 00007f4d17f7848c 00007f4d16840a84 00007f4d16840a11 00007f4d16840924 00007f4d110ecce9 00007f4d110ec632 00007f4d110edb8c 00007f4d110efa69 00007f4d110ef9ba 00007f4d110ef943
0619 00:12:00.783656 (+ 70805us) inbound_call.cc:162] Queueing success response
Metrics: {"spinlock_wait_cycles":4404071680}
I0619 00:12:00.794353 25068 catalog_manager.cc:2891] Scheduling retry of a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC with a delay of 291 ms (attempt = 5)
W0619 00:12:00.794613 25068 catalog_manager.cc:2910] Async tablet task a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a4c3d22a3237408387518e3061470d01 consensus state has no leader
W0619 00:12:00.794780 26009 rpcz_store.cc:255] Trace:
0619 00:11:59.947285 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:11:59.951040 (+  3755us) service_pool.cc:221] Handling call
0619 00:12:00.698857 (+747817us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b4c00000f20. stack: 00007f4d0f86d614 00007f4d0f86d510 00007f4d0f2471b4 00007f4d17f43074 00007f4d17f43009 00007f4d17f19e9c 00007f4d17f7848c 00007f4d16840a84 00007f4d16840a11 00007f4d16840924 00007f4d110ecce9 00007f4d110ec632 00007f4d110edb8c 00007f4d110efa69 00007f4d110ef9ba 00007f4d110ef943
0619 00:12:00.785634 (+ 86777us) inbound_call.cc:162] Queueing success response
Metrics: {"spinlock_wait_cycles":1718936320}
I0619 00:12:00.787273 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 353) took 5028ms. Request Metrics: {"tablet-delete.queue_time_us":5027845}
I0619 00:12:00.796676 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 135 ms (attempt = 4)
W0619 00:12:00.796948 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.801167 26018 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" instance_seqno: 1529392305884030) as {username='jenkins'} at 127.24.119.1:54954; Asking this server to re-register.
W0619 00:12:00.801802 26018 rpcz_store.cc:251] Call kudu.master.MasterService.TSHeartbeat from 127.24.119.1:54954 (request call id 40) took 2202ms (client timeout 1000).
W0619 00:12:00.802733 26018 rpcz_store.cc:255] Trace:
0619 00:11:58.599396 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:11:58.599618 (+   222us) service_pool.cc:221] Handling call
0619 00:12:00.800481 (+2200863us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b4c00000f20. stack: 00007f4d0f86d614 00007f4d0f86d510 00007f4d0f2471b4 00007f4d17f43074 00007f4d17f43009 00007f4d17f19e9c 00007f4d17f7848c 00007f4d16840a84 00007f4d16840a11 00007f4d16840924 00007f4d110ecce9 00007f4d110ec632 00007f4d110edb8c 00007f4d110efa69 00007f4d110ef9ba 00007f4d110ef943
0619 00:12:00.801759 (+  1278us) inbound_call.cc:162] Queueing success response
Metrics: {"spinlock_wait_cycles":5055044480}
I0619 00:12:00.803686 25730 ts_tablet_manager.cc:1382] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:00.801237 26012 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" instance_seqno: 1529392305884030) as {username='jenkins'} at 127.24.119.1:54954; Asking this server to re-register.
W0619 00:12:00.804060 26012 rpcz_store.cc:251] Call kudu.master.MasterService.TSHeartbeat from 127.24.119.1:54954 (request call id 42) took 975ms (client timeout 1000).
I0619 00:12:00.804467 25087 catalog_manager.cc:4025] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: Sending DeleteTablet for 3 replicas of tablet 14179adec4b241be87f42625f5edefc6
I0619 00:12:00.804803 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14179adec4b241be87f42625f5edefc6 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.805215 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14179adec4b241be87f42625f5edefc6 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:12:00 PDT)
W0619 00:12:00.805492 26012 rpcz_store.cc:255] Trace:
0619 00:11:59.828539 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:11:59.828681 (+   142us) service_pool.cc:221] Handling call
0619 00:12:00.800970 (+972289us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b4c00000f20. stack: 00007f4d0f86d614 00007f4d0f86d510 00007f4d0f2471b4 00007f4d17f43074 00007f4d17f43009 00007f4d17f19e9c 00007f4d17f7848c 00007f4d16840a84 00007f4d16840a11 00007f4d16840924 00007f4d110ecce9 00007f4d110ec632 00007f4d110edb8c 00007f4d110efa69 00007f4d110ef9ba 00007f4d110ef943
0619 00:12:00.804017 (+  3047us) inbound_call.cc:162] Queueing success response
Metrics: {"spinlock_wait_cycles":2308043520}
I0619 00:12:00.801326 26010 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "395763cf325d4ce19859830a7ad35c85" instance_seqno: 1529392308087416) as {username='jenkins'} at 127.24.119.3:37305; Asking this server to re-register.
W0619 00:12:00.806833 26010 rpcz_store.cc:251] Call kudu.master.MasterService.TSHeartbeat from 127.24.119.3:37305 (request call id 24) took 945ms (client timeout 1000).
I0619 00:12:00.797421 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d0624684b6b04ba3be260b2705dc00e1" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.797827 25462 tablet_service.cc:759] Processing CreateTablet for tablet 687c893ccc6346debb555de59d3e5341 (table=default.table_defc57494cc44555be460e4b77d34fe8 [id=f7a57d952d8a4cdf8c7ecdf22e85df45]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:00.807448 26010 rpcz_store.cc:255] Trace:
0619 00:11:59.861185 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:11:59.861279 (+    94us) service_pool.cc:221] Handling call
0619 00:12:00.800493 (+939214us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b4c00000f20. stack: 00007f4d0f86d614 00007f4d0f86d510 00007f4d0f2471b4 00007f4d17f43074 00007f4d17f43009 00007f4d17f19e9c 00007f4d17f7848c 00007f4d16840a84 00007f4d16840a11 00007f4d16840924 00007f4d110ecce9 00007f4d110ec632 00007f4d110edb8c 00007f4d110efa69 00007f4d110ef9ba 00007f4d110ef943
0619 00:12:00.806791 (+  6298us) inbound_call.cc:162] Queueing success response
Metrics: {"spinlock_wait_cycles":2207197056}
I0619 00:12:00.807548 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 687c893ccc6346debb555de59d3e5341. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.808821 25725 ts_tablet_manager.cc:1028] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:12:00.809309 25725 tablet_bootstrap.cc:427] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:12:00.809835 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14179adec4b241be87f42625f5edefc6 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.810016 25730 log.cc:986] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/e9a221a1d9814ef99a0b4ccaa04d1017
W0619 00:12:00.805976 25067 catalog_manager.cc:2910] Async tablet task 2a0b025496ee4087acadc27bbb6d4983 (table default.table_ed122503385b4360827d3299026dc91e [id=8452c371978c48f28ea44cb0fe5293ec]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a0b025496ee4087acadc27bbb6d4983 consensus state has no leader
I0619 00:12:00.806330 25071 catalog_manager.cc:2891] Scheduling retry of 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC with a delay of 78 ms (attempt = 2)
W0619 00:12:00.810696 25071 catalog_manager.cc:2910] Async tablet task 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 894de151fa3b4a4fbc962a5cafbccd3a consensus state has no leader
I0619 00:12:00.813191 25730 ts_tablet_manager.cc:1403] T e9a221a1d9814ef99a0b4ccaa04d1017 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:12:00.813820 26339 tablet.cc:1200] T 71510e2112fd4dc9ba08de08648d7ad7 P c007a00e65ad41a79da1a0c1050c55b4: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:00.814211 25069 catalog_manager.cc:2891] Scheduling retry of 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC with a delay of 78 ms (attempt = 3)
W0619 00:12:00.814471 25069 catalog_manager.cc:2910] Async tablet task 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 24281f894abb44419f18834dca2de65e consensus state has no leader
I0619 00:12:00.816020 25069 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 161 ms (attempt = 4)
W0619 00:12:00.816253 25069 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.818141 25602 tablet_service.cc:800] Processing DeleteTablet for tablet ef1566aa89b24620af70f2322df0dd56 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.818574 25334 tablet_service.cc:800] Processing DeleteTablet for tablet ef1566aa89b24620af70f2322df0dd56 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.818923 25331 tablet_service.cc:800] Processing DeleteTablet for tablet 14179adec4b241be87f42625f5edefc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:12:00.819432 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 0474be659e5e4a4cbfc181b8c812b1eb because the tablet was not found. No further retry: Not found: Tablet not found: 0474be659e5e4a4cbfc181b8c812b1eb
W0619 00:12:00.819939 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a4c653a255984d3bac88df63362b9334 because the tablet was not found. No further retry: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
I0619 00:12:00.817312 25724 tablet_bootstrap.cc:427] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:00.820595 25724 ts_tablet_manager.cc:1045] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.197s	user 0.007s	sys 0.003s
W0619 00:12:00.826318 25290 leader_election.cc:306] T f1b566b2b58f4955aa25ae990a4a8a1a P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: f1b566b2b58f4955aa25ae990a4a8a1a
I0619 00:12:00.804802 25736 ts_tablet_manager.cc:1382] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:00.830566 25736 log.cc:986] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/9d44681acacf4f04b6fcfbd4c9a18e87
I0619 00:12:00.831179 25736 ts_tablet_manager.cc:1403] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:12:00.831722 25087 catalog_manager.cc:3663] Got report from deleted tablet 1c4c69294cb945938004e5b9a0550e72 (table default.table_297b8bff60264a579f026fbd82396f53 [id=9c66376e1d444a928cdf40dbb2292f5f]) (Table deleted at 2018-06-19 00:11:59 PDT): Sending delete request for this tablet
I0619 00:12:00.832583 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1c4c69294cb945938004e5b9a0550e72 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:59 PDT)
W0619 00:12:00.833889 25290 consensus_peers.cc:434] T b34043f14f1c4edd97a1d4689ced5b32 P c007a00e65ad41a79da1a0c1050c55b4 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Couldn't send request to peer 395763cf325d4ce19859830a7ad35c85 for tablet b34043f14f1c4edd97a1d4689ced5b32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b34043f14f1c4edd97a1d4689ced5b32. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:00.834816 25402 heartbeater.cc:426] Registering TS with master...
I0619 00:12:00.821310 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 14179adec4b241be87f42625f5edefc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.836486 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 355) took 5011ms. Request Metrics: {"tablet-delete.queue_time_us":4967738}
I0619 00:12:00.837249 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 356) took 5007ms. Request Metrics: {"tablet-delete.queue_time_us":5005874}
I0619 00:12:00.837868 25741 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.1:46159 (request call id 358) took 4997ms. Request Metrics: {"tablet-delete.queue_time_us":4996052}
I0619 00:12:00.838389 25741 tablet_replica.cc:261] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: stopping tablet replica
I0619 00:12:00.817382 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 264) took 4946ms. Request Metrics: {"tablet-delete.queue_time_us":4672179}
I0619 00:12:00.839195 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 267) took 4825ms. Request Metrics: {"tablet-delete.queue_time_us":4824867}
I0619 00:12:00.840045 25730 tablet_replica.cc:261] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
W0619 00:12:00.828886 25558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:38512, user_credentials={real_user=jenkins}} blocked reactor thread for 48591.2us
I0619 00:12:00.842975 25668 heartbeater.cc:426] Registering TS with master...
I0619 00:12:00.843156 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(756057a7c34a4c61a261deaf849e433a)
I0619 00:12:00.843430 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(756057a7c34a4c61a261deaf849e433a)
I0619 00:12:00.843685 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(756057a7c34a4c61a261deaf849e433a)
I0619 00:12:00.843924 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(756057a7c34a4c61a261deaf849e433a)
I0619 00:12:00.844189 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(756057a7c34a4c61a261deaf849e433a)
I0619 00:12:00.844557 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(756057a7c34a4c61a261deaf849e433a)
I0619 00:12:00.844806 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(756057a7c34a4c61a261deaf849e433a)
I0619 00:12:00.845067 25730 raft_consensus.cc:2042] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.845347 25730 raft_consensus.cc:2069] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.793179 25727 tablet_bootstrap.cc:578] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4: No blocks or log segments found. Creating new log.
I0619 00:12:00.846011 26313 raft_consensus.cc:457] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10bda0b4856f4441b5aed24cdc4764a8)
I0619 00:12:00.846316 26313 raft_consensus.cc:2732] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Snoozing failure detection for 2.128s (starting election)
I0619 00:12:00.847010 26313 raft_consensus.cc:479] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.847925 26313 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.850790 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op CompactRowSetsOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:12:00.851068 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MinorDeltaCompactionOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:12:00.851295 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op MajorDeltaCompactionOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:12:00.851521 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op UndoDeltaBlockGCOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:12:00.851800 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushMRSOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:12:00.852053 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op FlushDeltaMemStoresOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:12:00.852468 25741 maintenance_manager.cc:237] P c007a00e65ad41a79da1a0c1050c55b4: Unregistered op LogGCOp(30a9d9fda8d34caeb8242c372439321b)
I0619 00:12:00.852841 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet e9a221a1d9814ef99a0b4ccaa04d1017 (table default.table_7846517150f341d0b6fc8df41c79b3a0 [id=6357deac871844f5b6f02f359c6a4357]) successfully deleted
W0619 00:12:00.853307 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 075076aaa16e4ea8a596a7a8c52f103e because the tablet was not found. No further retry: Not found: Tablet not found: 075076aaa16e4ea8a596a7a8c52f103e
I0619 00:12:00.853991 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 1c4c69294cb945938004e5b9a0550e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:59 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.853368 25741 raft_consensus.cc:2042] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:00.855849 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:00.856155 25536 heartbeater.cc:426] Registering TS with master...
I0619 00:12:00.856380 25486 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.714s (vote granted)
I0619 00:12:00.856637 25486 raft_consensus.cc:2245] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 1.
W0619 00:12:00.857075 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 8e3a356933e749b9a4ade476ec30f475 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 8e3a356933e749b9a4ade476ec30f475 already in progress: creating tablet
I0619 00:12:00.857197 25463 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:00.857333 25067 catalog_manager.cc:2891] Scheduling retry of 8e3a356933e749b9a4ade476ec30f475 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 21 ms (attempt = 1)
W0619 00:12:00.857913 25067 catalog_manager.cc:3083] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet 3e7c4b61f5314e40ad71371155a50b05 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 3e7c4b61f5314e40ad71371155a50b05 already in progress: creating tablet
I0619 00:12:00.858140 25067 catalog_manager.cc:2891] Scheduling retry of 3e7c4b61f5314e40ad71371155a50b05 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 19 ms (attempt = 1)
W0619 00:12:00.858530 25067 catalog_manager.cc:3071] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): delete failed for tablet a4c653a255984d3bac88df63362b9334 because the tablet was not found. No further retry: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
I0619 00:12:00.858739 25463 tablet_service.cc:759] Processing CreateTablet for tablet 24281f894abb44419f18834dca2de65e (table=default.table_4eb928c7853a4940ba9a83ba7f30a181 [id=388ad92a190b457497ffa5304c077739]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:00.859825 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 24281f894abb44419f18834dca2de65e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:00.861966 25462 ts_tablet_manager.cc:1173] T 687c893ccc6346debb555de59d3e5341 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:00.862437 25558 leader_election.cc:306] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.862751 25558 leader_election.cc:270] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:00.863206 26223 raft_consensus.cc:2732] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.703s (election complete)
I0619 00:12:00.840342 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 396) took 3828ms. Request Metrics: {"mutex_wait_us":7534,"tablet-delete.queue_time_us":3557789}
I0619 00:12:00.863514 26223 raft_consensus.cc:2487] T d0624684b6b04ba3be260b2705dc00e1 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:00.863728 25741 raft_consensus.cc:2069] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:00.838361 25724 raft_consensus.cc:325] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.864228 26010 ts_manager.cc:81] Registered new tserver with Master: 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.864431 25724 raft_consensus.cc:351] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.865238 25724 raft_consensus.cc:628] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:00.865999 25724 consensus_queue.cc:227] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.872557 26313 leader_election.cc:243] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:00.873313 25730 ts_tablet_manager.cc:1369] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.877257 25052 master-stress-test.cc:468] Killing test thread 1/10
I0619 00:12:00.880388 26223 raft_consensus.cc:457] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:00.880628 26223 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.929s (starting election)
I0619 00:12:00.880964 26223 raft_consensus.cc:479] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.881989 26223 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.882814 26223 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:00.883432 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7c4b61f5314e40ad71371155a50b05 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:12:00.883828 25351 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb3c60fa2d8f4cafa12a5b0bc806433f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c007a00e65ad41a79da1a0c1050c55b4" is_pre_election: true
I0619 00:12:00.884042 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
W0619 00:12:00.885064 25553 leader_election.cc:306] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:00.890993 25067 catalog_manager.cc:2891] Scheduling retry of a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC with a delay of 549 ms (attempt = 6)
W0619 00:12:00.891265 25067 catalog_manager.cc:2910] Async tablet task a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a7a3d73c048749b489660d109fa15158 consensus state has no leader
I0619 00:12:00.891912 25071 catalog_manager.cc:2891] Scheduling retry of 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC with a delay of 97 ms (attempt = 3)
W0619 00:12:00.892199 25071 catalog_manager.cc:2910] Async tablet task 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 894de151fa3b4a4fbc962a5cafbccd3a consensus state has no leader
I0619 00:12:00.893645 25069 catalog_manager.cc:2891] Scheduling retry of 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC with a delay of 170 ms (attempt = 4)
W0619 00:12:00.893906 25069 catalog_manager.cc:2910] Async tablet task 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 24281f894abb44419f18834dca2de65e consensus state has no leader
I0619 00:12:00.888600 25741 ts_tablet_manager.cc:1369] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:00.905869 25071 catalog_manager.cc:2891] Scheduling retry of 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC with a delay of 560 ms (attempt = 6)
W0619 00:12:00.906117 25071 catalog_manager.cc:2910] Async tablet task 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 455d8e1ee5be484b8005312d2d886821 consensus state has no leader
I0619 00:12:00.908355 25068 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 1057 ms (attempt = 7)
W0619 00:12:00.908648 25068 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:00.912111 25334 tablet_service.cc:800] Processing DeleteTablet for tablet 3e7c4b61f5314e40ad71371155a50b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.1:46159
I0619 00:12:00.912717 25334 tablet_service.cc:800] Processing DeleteTablet for tablet 8e3a356933e749b9a4ade476ec30f475 with delete_type TABLET_DATA_DELETED (replaced by tablet beddf1523a854f5b9292c0e4013d7436) from {username='jenkins'} at 127.24.119.1:46159
W0619 00:12:00.912818 25290 leader_election.cc:306] T 30a9d9fda8d34caeb8242c372439321b P c007a00e65ad41a79da1a0c1050c55b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 30a9d9fda8d34caeb8242c372439321b
I0619 00:12:00.913853 26010 ts_manager.cc:81] Registered new tserver with Master: c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:00.922096 25067 catalog_manager.cc:2891] Scheduling retry of 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC with a delay of 2048 ms (attempt = 8)
W0619 00:12:00.922369 25067 catalog_manager.cc:2910] Async tablet task 4492e3339ba149a5bbd2f581eea6b7ef (table default.table_9dfa98569b4d490895c509806ae31ce6 [id=e3fb21dbab7c4411827836d3683a5480]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 4492e3339ba149a5bbd2f581eea6b7ef consensus state has no leader
I0619 00:12:00.923070 25086 master_service.cc:523] ReplaceTablet: received request to replace tablet 1b40a55533f74f5eb90497d3c1064034 from {username='jenkins'} at 127.24.119.62:54390
I0619 00:12:00.929936 25727 tablet_bootstrap.cc:427] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4: No bootstrap required, opened a new log
I0619 00:12:00.930511 25727 ts_tablet_manager.cc:1045] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4: Time spent bootstrapping tablet: real 0.155s	user 0.007s	sys 0.001s
I0619 00:12:00.932574 25067 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 1042 ms (attempt = 7)
W0619 00:12:00.932911 25067 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:12:00.933302 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 272 ms (attempt = 5)
I0619 00:12:00.932410 25727 raft_consensus.cc:325] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:12:00.933537 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.934095 25724 ts_tablet_manager.cc:1073] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.097s	user 0.004s	sys 0.000s
I0619 00:12:00.934476 25724 ts_tablet_manager.cc:1028] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:00.866777 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 395) took 3900ms. Request Metrics: {"tablet-delete.queue_time_us":3795388}
I0619 00:12:00.937247 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 399) took 3856ms. Request Metrics: {"tablet-delete.queue_time_us":3855204}
I0619 00:12:00.940248 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 400) took 3856ms. Request Metrics: {"tablet-delete.queue_time_us":3854625}
I0619 00:12:00.942502 25724 tablet_bootstrap.cc:427] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:00.942860 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 4142 ms (attempt = 9)
W0619 00:12:00.943162 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:12:00.946012 25724 tablet_bootstrap.cc:578] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:00.949287 25727 raft_consensus.cc:351] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:00.949591 25727 raft_consensus.cc:628] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c007a00e65ad41a79da1a0c1050c55b4, State: Initialized, Role: FOLLOWER
I0619 00:12:00.950275 25727 consensus_queue.cc:227] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:00.956224 25052 master-stress-test.cc:470] Killed test thread 1/10
I0619 00:12:00.956408 25052 master-stress-test.cc:468] Killing test thread 2/10
I0619 00:12:00.956614 25052 master-stress-test.cc:470] Killed test thread 2/10
I0619 00:12:00.956755 25052 master-stress-test.cc:468] Killing test thread 3/10
I0619 00:12:00.956923 25052 master-stress-test.cc:470] Killed test thread 3/10
I0619 00:12:00.957062 25052 master-stress-test.cc:468] Killing test thread 4/10
I0619 00:12:00.957252 25052 master-stress-test.cc:470] Killed test thread 4/10
I0619 00:12:00.957391 25052 master-stress-test.cc:468] Killing test thread 5/10
I0619 00:12:00.957556 25052 master-stress-test.cc:470] Killed test thread 5/10
I0619 00:12:00.957726 25052 master-stress-test.cc:468] Killing test thread 6/10
I0619 00:12:00.957892 25052 master-stress-test.cc:470] Killed test thread 6/10
I0619 00:12:00.958022 25052 master-stress-test.cc:468] Killing test thread 7/10
I0619 00:12:00.958184 25052 master-stress-test.cc:470] Killed test thread 7/10
I0619 00:12:00.958312 25052 master-stress-test.cc:468] Killing test thread 8/10
I0619 00:12:00.958515 25052 master-stress-test.cc:470] Killed test thread 8/10
I0619 00:12:00.958680 25052 master-stress-test.cc:468] Killing test thread 9/10
I0619 00:12:00.959074 25052 master-stress-test.cc:470] Killed test thread 9/10
I0619 00:12:00.959218 25052 master-stress-test.cc:468] Killing test thread 10/10
I0619 00:12:00.959805 25069 catalog_manager.cc:2891] Scheduling retry of 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC with a delay of 2097 ms (attempt = 8)
W0619 00:12:00.960072 25069 catalog_manager.cc:2910] Async tablet task 248dd7541f7e4f43ba916a888bab98e1 (table default.table_034f2a79dabd489ba481a00fccab1d84 [id=44cd3f9c9bdc46348e2d00b5c26c1be7]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 248dd7541f7e4f43ba916a888bab98e1 consensus state has no leader
I0619 00:12:00.959507 25052 master-stress-test.cc:470] Killed test thread 10/10
I0619 00:12:00.960310 25727 ts_tablet_manager.cc:1073] T 6040e369e0fb445fb993d12dd883039b P c007a00e65ad41a79da1a0c1050c55b4: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.000s
I0619 00:12:00.960976 25727 ts_tablet_manager.cc:1028] T 1cfd335c18c047fba2facc5194807abb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrapping tablet
I0619 00:12:00.961544 25727 tablet_bootstrap.cc:427] T 1cfd335c18c047fba2facc5194807abb P c007a00e65ad41a79da1a0c1050c55b4: Bootstrap starting.
I0619 00:12:00.960554 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 25250
I0619 00:12:00.965266 26338 tablet.cc:1200] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:00.942641 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 401) took 3837ms. Request Metrics: {"tablet-delete.queue_time_us":3836480}
I0619 00:12:00.716428 26054 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.119.61:54208 (request call id 1) took 4175ms. Request Metrics: {"raft.queue_time_us":15188,"raft.run_cpu_time_us":7366,"raft.run_wall_time_us":2065346,"spinlock_wait_cycles":4936651008,"thread_start_us":44223,"threads_started":2,"wal-append.queue_time_us":27910,"child_traces":[["txn",{"apply.queue_time_us":47063,"apply.run_cpu_time_us":1152,"apply.run_wall_time_us":4037,"num_ops":1,"prepare.queue_time_us":94285,"prepare.run_cpu_time_us":670,"prepare.run_wall_time_us":754,"replication_time_us":1723820,"thread_start_us":144733,"threads_started":2}],["txn",{"apply.queue_time_us":3618,"apply.run_cpu_time_us":754,"apply.run_wall_time_us":754,"num_ops":2,"prepare.queue_time_us":525,"prepare.run_cpu_time_us":666,"prepare.run_wall_time_us":1024,"replication_time_us":1648977,"thread_start_us":20060,"threads_started":1}],["txn",{"apply.queue_time_us":2169,"apply.run_cpu_time_us":854,"apply.run_wall_time_us":854,"num_ops":2,"prepare.queue_time_us":2132,"prepare.run_cpu_time_us":594,"prepare.run_wall_time_us":594,"replication_time_us":1665626}],["txn",{"apply.queue_time_us":2742,"apply.run_cpu_time_us":572,"apply.run_wall_time_us":864,"num_ops":1,"prepare.queue_time_us":3783,"prepare.run_cpu_time_us":465,"prepare.run_wall_time_us":465,"replication_time_us":1658017}],["txn",{"apply.queue_time_us":3541,"apply.run_cpu_time_us":639,"apply.run_wall_time_us":700,"num_ops":1,"prepare.queue_time_us":3937,"prepare.run_cpu_time_us":475,"prepare.run_wall_time_us":475,"replication_time_us":1658020,"thread_start_us":50271,"threads_started":1}],["txn",{"apply.queue_time_us":6278,"apply.run_cpu_time_us":1173,"apply.run_wall_time_us":5666,"num_ops":2,"prepare.queue_time_us":4016,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":613,"replication_time_us":1708082}],["txn",{"apply.queue_time_us":4605,"apply.run_cpu_time_us":1083,"apply.run_wall_time_us":1083,"num_ops":1,"prepare.queue_time_us":4337,"prepare.run_cpu_time_us":1047,"prepare.run_wall_time_us":1713933,"replication_time_us":1711098,"row_lock_wait_count":1,"row_lock_wait_us":1713258}],["txn",{"apply.queue_time_us":4960,"apply.run_cpu_time_us":802,"apply.run_wall_time_us":802,"num_ops":2,"prepare.queue_time_us":1718029,"prepare.run_cpu_time_us":707,"prepare.run_wall_time_us":707,"replication_time_us":1712083}],["txn",{"apply.queue_time_us":4943,"apply.run_cpu_time_us":1027,"apply.run_wall_time_us":3055,"num_ops":2,"prepare.queue_time_us":1718441,"prepare.run_cpu_time_us":9393,"prepare.run_wall_time_us":66832,"replication_time_us":1711798,"thread_start_us":66171,"threads_started":1}],["txn",{"apply.queue_time_us":1521,"apply.run_cpu_time_us":836,"apply.run_wall_time_us":836,"num_ops":1,"prepare.queue_time_us":1785011,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":1562,"replication_time_us":1711493}],["txn",{"apply.queue_time_us":1726,"apply.run_cpu_time_us":886,"apply.run_wall_time_us":885,"num_ops":2,"prepare.queue_time_us":1786300,"prepare.run_cpu_time_us":638,"prepare.run_wall_time_us":637,"replication_time_us":1711176}],["txn",{"apply.queue_time_us":1952,"apply.run_cpu_time_us":1362,"apply.run_wall_time_us":1363,"num_ops":2,"prepare.queue_time_us":1786636,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":4021,"replication_time_us":1710864}],["txn",{"apply.queue_time_us":803,"apply.run_cpu_time_us":703,"apply.run_wall_time_us":703,"num_ops":1,"prepare.queue_time_us":1790408,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":547,"replication_time_us":1710576}],["txn",{"apply.queue_time_us":706,"apply.run_cpu_time_us":749,"apply.run_wall_time_us":748,"num_ops":2,"prepare.queue_time_us":1790675,"prepare.run_cpu_time_us":671,"prepare.run_wall_time_us":671,"replication_time_us":1710281}],["txn",{"apply.queue_time_us":214,"apply.run_cpu_time_us":709,"apply.run_wall_time_us":712,"num_ops":1,"prepare.queue_time_us":1688837,"prepare.run_cpu_time_us":564,"prepare.run_wall_time_us":1460,"replication_time_us":1607807,"spinlock_wait_cycles":60928}],["txn",{"apply.queue_time_us":690,"apply.run_cpu_time_us":711,"apply.run_wall_time_us":712,"num_ops":2,"prepare.queue_time_us":1689992,"prepare.run_cpu_time_us":1105,"prepare.run_wall_time_us":1106,"replication_time_us":1607403}],["txn",{"apply.queue_time_us":902,"apply.run_cpu_time_us":571,"apply.run_wall_time_us":572,"num_ops":1,"prepare.queue_time_us":1690746,"prepare.run_cpu_time_us":497,"prepare.run_wall_time_us":2001,"replication_time_us":1607024}],["txn",{"apply.queue_time_us":54498,"apply.run_cpu_time_us":1005,"apply.run_wall_time_us":1087,"num_ops":2,"prepare.queue_time_us":1692430,"prepare.run_cpu_time_us":613,"prepare.run_wall_time_us":56377,"replication_time_us":1607192}],["txn",{"apply.queue_time_us":72,"apply.run_cpu_time_us":969,"apply.run_wall_time_us":969,"num_ops":2,"prepare.queue_time_us":1748508,"prepare.run_cpu_time_us":750,"prepare.run_wall_time_us":2183,"replication_time_us":1606879}],["txn",{"apply.queue_time_us":71,"apply.run_cpu_time_us":707,"apply.run_wall_time_us":706,"num_ops":1,"prepare.queue_time_us":1750423,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":1396,"replication_time_us":1606559}],["txn",{"apply.queue_time_us":17979,"apply.run_cpu_time_us":1070,"apply.run_wall_time_us":1069,"num_ops":2,"prepare.queue_time_us":1751547,"prepare.run_cpu_time_us":632,"prepare.run_wall_time_us":19834,"replication_time_us":1606269}],["txn",{"apply.queue_time_us":966,"apply.run_cpu_time_us":919,"apply.run_wall_time_us":950,"num_ops":2,"prepare.queue_time_us":1771068,"prepare.run_cpu_time_us":661,"prepare.run_wall_time_us":716,"replication_time_us":1605952}],["txn",{"apply.queue_time_us":1100,"apply.run_cpu_time_us":866,"apply.run_wall_time_us":866,"num_ops":2,"prepare.queue_time_us":1771546,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":19011,"replication_time_us":1605624}],["txn",{"apply.queue_time_us":3880,"apply.run_cpu_time_us":948,"apply.run_wall_time_us":948,"num_ops":2,"prepare.queue_time_us":1790258,"prepare.run_cpu_time_us":718,"prepare.run_wall_time_us":5614,"replication_time_us":1605355}],["txn",{"apply.queue_time_us":4124,"apply.run_cpu_time_us":719,"apply.run_wall_time_us":719,"num_ops":1,"prepare.queue_time_us":1763885,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":5484,"replication_time_us":1573323}],["txn",{"apply.queue_time_us":3720,"apply.run_cpu_time_us":634,"apply.run_wall_time_us":635,"num_ops":1,"prepare.queue_time_us":1768896,"prepare.run_cpu_time_us":551,"prepare.run_wall_time_us":4965,"replication_time_us":1572784}],["txn",{"apply.queue_time_us":969,"apply.run_cpu_time_us":1149,"apply.run_wall_time_us":128404,"num_ops":2,"prepare.queue_time_us":1773570,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":5254,"replication_time_us":1572449}],["txn",{"apply.queue_time_us":19862,"apply.run_cpu_time_us":1093,"apply.run_wall_time_us":108566,"mutex_wait_us":1366,"num_ops":1,"prepare.queue_time_us":1778531,"prepare.run_cpu_time_us":635,"prepare.run_wall_time_us":14524,"replication_time_us":1572112}],["txn",{"apply.queue_time_us":92,"apply.run_cpu_time_us":822,"apply.run_wall_time_us":821,"num_ops":2,"prepare.queue_time_us":1792800,"prepare.run_cpu_time_us":816,"prepare.run_wall_time_us":10067,"replication_time_us":1571816}],["txn",{"apply.queue_time_us":65,"apply.run_cpu_time_us":1053,"apply.run_wall_time_us":20023,"num_ops":2,"prepare.queue_time_us":1802568,"prepare.run_cpu_time_us":633,"prepare.run_wall_time_us":1031,"replication_time_us":1571495}],["txn",{"apply.queue_time_us":93,"apply.run_cpu_time_us":809,"apply.run_wall_time_us":807,"num_ops":1,"prepare.queue_time_us":1803311,"prepare.run_cpu_time_us":1920,"prepare.run_wall_time_us":115770,"replication_time_us":1571165,"row_lock_wait_count":1,"row_lock_wait_us":41110,"spinlock_wait_cycles":121085184}],["txn",{"apply.queue_time_us":63,"apply.run_cpu_time_us":894,"apply.run_wall_time_us":894,"num_ops":2,"prepare.queue_time_us":1918817,"prepare.run_cpu_time_us":711,"prepare.run_wall_time_us":1765,"replication_time_us":1570865}],["txn",{"apply.queue_time_us":88,"apply.run_cpu_time_us":849,"apply.run_wall_time_us":849,"num_ops":2,"prepare.queue_time_us":1839624,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":1612,"replication_time_us":1570525}],["txn",{"apply.queue_time_us":62,"apply.run_cpu_time_us":752,"apply.run_wall_time_us":753,"num_ops":2,"prepare.queue_time_us":1840827,"prepare.run_cpu_time_us":550,"prepare.run_wall_time_us":1402,"replication_time_us":1489475}],["txn",{"apply.queue_time_us":64,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":619,"num_ops":1,"prepare.queue_time_us":1841933,"prepare.run_cpu_time_us":539,"prepare.run_wall_time_us":1366,"replication_time_us":1489094}],["txn",{"apply.queue_time_us":67,"apply.run_cpu_time_us":716,"apply.run_wall_time_us":716,"num_ops":2,"prepare.queue_time_us":1843019,"prepare.run_cpu_time_us":687,"prepare.run_wall_time_us":1509,"replication_time_us":1488796}],["txn",{"apply.queue_time_us":83,"apply.run_cpu_time_us":610,"apply.run_wall_time_us":610,"num_ops":1,"prepare.queue_time_us":1844213,"prepare.run_cpu_time_us":523,"prepare.run_wall_time_us":1237,"replication_time_us":1488455}],["txn",{"num_ops":2,"prepare.queue_time_us":1845158,"replication_time_us":1488145}],["txn",{"replication_time_us":1487824}],["txn",{"replication_time_us":1487496}],["txn",{"replication_time_us":1487191}],["txn",{"replication_time_us":1486878}],["txn",{"replication_time_us":1459325}],["txn",{"replication_time_us":1459020}],["txn",{"replication_time_us":1458704}],["txn",{"replication_time_us":1458278}],["txn",{"replication_time_us":1457965}],["txn",{"replication_time_us":1457716}],["txn",{"replication_time_us":1457406}],["txn",{"replication_time_us":1457095}],["txn",{"replication_time_us":1456819}],["txn",{"replication_time_us":1456556}],["txn",{"replication_time_us":1456230}],["txn",{"replication_time_us":1455914}],["txn",{"replication_time_us":1422976}],["txn",{"replication_time_us":1422616}],["txn",{"replication_time_us":1422309}],["txn",{"replication_time_us":1422017}],["txn",{"replication_time_us":1421731}],["txn",{"replication_time_us":1421357}],["txn",{"replication_time_us":1421035}],["txn",{"replication_time_us":1400585}],["txn",{"replication_time_us":1400172}],["txn",{"replication_time_us":1399867}],["txn",{"replication_time_us":1399476}],["txn",{"replication_time_us":1399163}],["txn",{"replication_time_us":1398876}],["txn",{"replication_time_us":1398562}],["txn",{"replication_time_us":1371717}],["txn",{"replication_time_us":1371338}],["txn",{"replication_time_us":1371036}],["txn",{"replication_time_us":1370687}],["txn",{"replication_time_us":1370380}],["txn",{"replication_time_us":1367803}],["txn",{"replication_time_us":1367387}],["txn",{"replication_time_us":1367032}],["txn",{"replication_time_us":1366714}],["txn",{"replication_time_us":1366393}],["txn",{"replication_time_us":1366041}],["txn",{"replication_time_us":1386978}],["txn",{"replication_time_us":1386683}],["txn",{"replication_time_us":1386377}],["txn",{"replication_time_us":1362936}],["txn",{"replication_time_us":1327249}],["txn",{"replication_time_us":1326919}],["txn",{"replication_time_us":1326650}],["txn",{"replication_time_us":1326361}],["txn",{"replication_time_us":1326041}],["txn",{"replication_time_us":1325746}],["txn",{"replication_time_us":1325456}],["txn",{"replication_time_us":1325149}],["txn",{"replication_time_us":1324840}],["txn",{"replication_time_us":1324564}],["txn",{"replication_time_us":1289309}],["txn",{"replication_time_us":1288914}],["txn",{"replication_time_us":1288556}],["txn",{"replication_time_us":1288253}],["txn",{"replication_time_us":1287940}],["txn",{"replication_time_us":1287525}],["txn",{"replication_time_us":1287209}],["txn",{"replication_time_us":1286908}],["txn",{"replication_time_us":1267780}],["txn",{"replication_time_us":1267474}],["txn",{"replication_time_us":1267153}],["txn",{"replication_time_us":1265496}],["txn",{"replication_time_us":1265196}],["txn",{"replication_time_us":1264869}],["txn",{"replication_time_us":1262967}],["txn",{"replication_time_us":1262690}],["txn",{"replication_time_us":1262371}],["txn",{"replication_time_us":1262077}],["txn",{"replication_time_us":1261793}],["txn",{"replication_time_us":1261500}],["txn",{"replication_time_us":1261187}],["txn",{"replication_time_us":1260887}],["txn",{"replication_time_us":1198843}],["txn",{"replication_time_us":1198473}],["txn",{"replication_time_us":1198179}],["txn",{"replication_time_us":1197911}],["txn",{"replication_time_us":1188068}],["txn",{"replication_time_us":1187733}],["txn",{"replication_time_us":1187450}],["txn",{"replication_time_us":1187150}],["txn",{"replication_time_us":1186817}],["txn",{"replication_time_us":1186500}],["txn",{"replication_time_us":1167937}],["txn",{"replication_time_us":1167568}],["txn",{"replication_time_us":1167209}],["txn",{"replication_time_us":1166797}],["txn",{"replication_time_us":1166353}],["txn",{"replication_time_us":1166042}],["txn",{"replication_time_us":1165311}],["txn",{"replication_time_us":1165047}],["txn",{"replication_time_us":1164788}],["txn",{"replication_time_us":1146576}],["txn",{"replication_time_us":1146229}],["txn",{"replication_time_us":1216228}],["txn",{"replication_time_us":1215896}],["txn",{"replication_time_us":1215591}],["txn",{"replication_time_us":1215244}],["txn",{"replication_time_us":1214945}],["txn",{"replication_time_us":1214620}],["txn",{"replication_time_us":1214257}],["txn",{"replication_time_us":1213938}],["txn",{"replication_time_us":1213635}],["txn",{"replication_time_us":1190282}],["txn",{"replication_time_us":1189927}],["txn",{"replication_time_us":1189049}],["txn",{"replication_time_us":1188680}],["txn",{"replication_time_us":1188375}],["txn",{"replication_time_us":1188068}],["txn",{"replication_time_us":1187743}],["txn",{"replication_time_us":1187377}],["txn",{"replication_time_us":1187031}],["txn",{"replication_time_us":1186663}],["txn",{"replication_time_us":1186299}],["txn",{"replication_time_us":1185833}],["txn",{"replication_time_us":1185514}],["txn",{"replication_time_us":1185174}],["txn",{"replication_time_us":1165536}],["txn",{"replication_time_us":1165091}],["txn",{"replication_time_us":1164779}],["txn",{"replication_time_us":1164364}],["txn",{"replication_time_us":1164048}],["txn",{"replication_time_us":1163695}],["txn",{"replication_time_us":1163379}],["txn",{"replication_time_us":1163056}],["txn",{"replication_time_us":1162591}],["txn",{"replication_time_us":1162332}],["txn",{"replication_time_us":1161997}],["txn",{"replication_time_us":1161655}],["txn",{"replication_time_us":1161314}],["txn",{"replication_time_us":1161014}],["txn",{"replication_time_us":1138137}],["txn",{"replication_time_us":1137770}],["txn",{"replication_time_us":1137474}],["txn",{"replication_time_us":1122951}],["txn",{"replication_time_us":1122638}],["txn",{"replication_time_us":1124667}],["txn",{"replication_time_us":1124384}],["txn",{"replication_time_us":1123724}],["txn",{"replication_time_us":1123457}],["txn",{"replication_time_us":1097497}],["txn",{"replication_time_us":1097163}],["txn",{"replication_time_us":1096864}],["txn",{"replication_time_us":1096608}],["txn",{"replication_time_us":1096309}],["txn",{"replication_time_us":1075835}],["txn",{"replication_time_us":1075535}],["txn",{"replication_time_us":1061910}],["txn",{"replication_time_us":1061545}],["txn",{"replication_time_us":1061195}],["txn",{"replication_time_us":1052167}],["txn",{"replication_time_us":1051842}],["txn",{"replication_time_us":1042280}],["txn",{"replication_time_us":1040250}],["txn",{"replication_time_us":1036843}],["txn",{"replication_time_us":1036330}],["txn",{"replication_time_us":1025985}],["txn",{"replication_time_us":1024669}],["txn",{"replication_time_us":1024366}],["txn",{"replication_time_us":1024036}],["txn",{"replication_time_us":1023744}],["txn",{"replication_time_us":1023473}],["txn",{"replication_time_us":1023182}],["txn",{"replication_time_us":1022895}],["txn",{"replication_time_us":1022627}],["txn",{"replication_time_us":1022394}],["txn",{"replication_time_us":1022149}],["txn",{"replication_time_us":1021872}],["txn",{"replication_time_us":989418}],["txn",{"replication_time_us":988043}],["txn",{"replication_time_us":987446}],["txn",{"replication_time_us":958513}],["txn",{"replication_time_us":958088}],["txn",{"replication_time_us":957739}],["txn",{"replication_time_us":957466}],["txn",{"replication_time_us":957153}],["txn",{"replication_time_us":956816}],["txn",{"replication_time_us":921124}],["txn",{"replication_time_us":920762}],["txn",{"replication_time_us":920426}],["txn",{"replication_time_us":920107}],["txn",{"replication_time_us":919807}],["txn",{"replication_time_us":919502}],["txn",{"replication_time_us":919152}],["txn",{"replication_time_us":918846}],["txn",{"replication_time_us":874649}],["txn",{"replication_time_us":874329}],["txn",{"replication_time_us":873992}],["txn",{"replication_time_us":873269}],["txn",{"replication_time_us":872962}],["txn",{"replication_time_us":872694}],["txn",{"replication_time_us":846441}],["txn",{"replication_time_us":845895}],["txn",{"replication_time_us":845482}],["txn",{"replication_time_us":845195}],["txn",{"replication_time_us":844912}],["txn",{"replication_time_us":844613}],["txn",{"replication_time_us":844331}],["txn",{"replication_time_us":844024}],["txn",{"replication_time_us":843687}],["txn",{"replication_time_us":843403}],["txn",{"replication_time_us":843121}],["txn",{"replication_time_us":816143}],["txn",{"replication_time_us":815760}],["txn",{"replication_time_us":815451}],["txn",{"replication_time_us":815122}],["txn",{"replication_time_us":814805}],["txn",{"replication_time_us":814485}],["txn",{"replication_time_us":814148}],["txn",{"replication_time_us":813836}],["txn",{"replication_time_us":813539}],["txn",{"replication_time_us":780592}],["txn",{"replication_time_us":780262}],["txn",{"replication_time_us":779526}],["txn",{"replication_time_us":779054}],["txn",{"replication_time_us":727488}],["txn",{"replication_time_us":727135}],["txn",{"replication_time_us":726808}],["txn",{"replication_time_us":726437}],["txn",{"replication_time_us":726143}],["txn",{"replication_time_us":725841}],["txn",{"replication_time_us":725515}],["txn",{"replication_time_us":725216}],["txn",{"replication_time_us":724882}],["txn",{"replication_time_us":724578}],["txn",{"replication_time_us":692203}],["txn",{"replication_time_us":691842}],["txn",{"replication_time_us":691484}],["txn",{"replication_time_us":691134}],["txn",{"replication_time_us":690650}],["txn",{"replication_time_us":679536}],["txn",{"replication_time_us":679216}],["txn",{"replication_time_us":678847}],["txn",{"replication_time_us":678529}],["txn",{"replication_time_us":678241}],["txn",{"replication_time_us":677856}],["txn",{"replication_time_us":677497}],["txn",{"replication_time_us":677192}],["txn",{"replication_time_us":676914}],["txn",{"replication_time_us":676593}],["txn",{"replication_time_us":676262}],["txn",{"replication_time_us":675914}],["txn",{"replication_time_us":675576}],["txn",{"replication_time_us":626203}],["txn",{"replication_time_us":625749}],["txn",{"replication_time_us":625248}],["txn",{"replication_time_us":624925}],["txn",{"replication_time_us":624603}],["txn",{"replication_time_us":624242}],["txn",{"replication_time_us":623759}],["txn",{"replication_time_us":623422}],["txn",{"replication_time_us":623102}],["txn",{"replication_time_us":622710}],["txn",{"replication_time_us":713725}],["txn",{"replication_time_us":713837}],["txn",{"replication_time_us":682485}],["txn",{}]]}
I0619 00:12:00.967273 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 402) took 3859ms. Request Metrics: {"tablet-delete.queue_time_us":3840995}
I0619 00:12:00.977653 25069 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 293 ms (attempt = 5)
W0619 00:12:00.977964 25069 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:00.980675 25087 catalog_manager.cc:3663] Got report from deleted tablet 14179adec4b241be87f42625f5edefc6 (table default.table_d036ca047e034c39958950ab500d6004 [id=6f8996225b924c6db7ebdeaebb35340e]) (Table deleted at 2018-06-19 00:12:00 PDT): Sending delete request for this tablet
I0619 00:12:00.981387 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14179adec4b241be87f42625f5edefc6 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.983247 25085 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:12:00.983623 25085 catalog_manager.cc:3881] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) does not have the latest schema for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]). Expected version 2 got 0
I0619 00:12:00.984519 25085 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:12:00.975124 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 403) took 3843ms. Request Metrics: {"tablet-delete.queue_time_us":3837730}
I0619 00:12:00.984921 25085 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 37 ms (attempt = 1)
W0619 00:12:00.985153 25085 catalog_manager.cc:3945] Failed to send 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:12:00.985556 25087 catalog_manager.cc:3663] Got report from deleted tablet 14179adec4b241be87f42625f5edefc6 (table default.table_d036ca047e034c39958950ab500d6004 [id=6f8996225b924c6db7ebdeaebb35340e]) (Table deleted at 2018-06-19 00:12:00 PDT): Sending delete request for this tablet
I0619 00:12:00.986210 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14179adec4b241be87f42625f5edefc6 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:00.986047 25086 catalog_manager.cc:3786] T 41e052fdd92b4c3d9803fa43433e60d5 P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:00.987045 25086 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]). Expected version 2 got 0
I0619 00:12:00.985106 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 404) took 3847ms. Request Metrics: {"tablet-delete.queue_time_us":3839523}
I0619 00:12:00.991015 25725 tablet_bootstrap.cc:578] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:12:00.988942 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 405) took 3827ms. Request Metrics: {"tablet-delete.queue_time_us":3809358}
I0619 00:12:00.992849 25736 tablet_replica.cc:261] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:12:00.992813 25071 catalog_manager.cc:2891] Scheduling retry of 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC with a delay of 131 ms (attempt = 4)
I0619 00:12:00.993088 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(3722b203a516466489c74ef6013eacae)
W0619 00:12:00.993155 25071 catalog_manager.cc:2910] Async tablet task 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 894de151fa3b4a4fbc962a5cafbccd3a consensus state has no leader
I0619 00:12:00.993293 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(3722b203a516466489c74ef6013eacae)
I0619 00:12:00.989863 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 14179adec4b241be87f42625f5edefc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.993484 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(3722b203a516466489c74ef6013eacae)
I0619 00:12:00.990089 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 14179adec4b241be87f42625f5edefc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:00.993665 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(3722b203a516466489c74ef6013eacae)
I0619 00:12:00.993911 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(3722b203a516466489c74ef6013eacae)
I0619 00:12:00.994112 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(3722b203a516466489c74ef6013eacae)
I0619 00:12:00.994292 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(3722b203a516466489c74ef6013eacae)
I0619 00:12:00.994459 25736 raft_consensus.cc:2042] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:00.994655 25736 consensus_peers.cc:456] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:12:00.994870 25736 consensus_peers.cc:456] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:12:00.995321 25736 raft_consensus.cc:2069] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:01.001921 25600 tablet_service.cc:759] Processing CreateTablet for tablet fbda888269cd4e25ba87671fcdeb2088 (table=default.table_38a9cc3b684f4e64a71452318b1dfdb1 [id=47a54f152b254567ba2857b5432f5dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.002537 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fbda888269cd4e25ba87671fcdeb2088. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:01.002774 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 144524us
I0619 00:12:01.005004 25602 tablet_service.cc:759] Processing CreateTablet for tablet 9160f06bedfd4b57bba18f7eab8d7e1f (table=default.table_89031abc2b4847ae883f7fb4ae304e9b [id=9b15de0b0f394a7d89f7d46bf3f50033]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.005553 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9160f06bedfd4b57bba18f7eab8d7e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.004804 26344 raft_consensus.cc:457] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.006481 26344 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.960s (starting election)
I0619 00:12:01.006744 26344 raft_consensus.cc:479] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.008486 26227 raft_consensus.cc:457] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.008785 26227 raft_consensus.cc:2732] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.804s (starting election)
I0619 00:12:01.008829 26344 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:01.009248 26344 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:01.009105 26227 raft_consensus.cc:479] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.010152 26227 leader_election.cc:243] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:01.010550 26227 leader_election.cc:243] T f155a418d0d547f7b775360f1a90ce01 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:01.022305 25255 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb25d1bba5bf4c64914f300f122ec898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25d1bba5bf4c64914f300f122ec898" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 37587 } } peers { permanent_uuid: "b2a016cc9f274bc1979068306ec6d15e" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 46325 } } peers { permanent_uuid: "8c39479445934adb9bdbb6e60c3bf712" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 57058 } } }
I0619 00:12:01.023069 25255 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:01.023783 26223 raft_consensus.cc:457] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.024113 26223 raft_consensus.cc:2732] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.706s (starting election)
I0619 00:12:01.024344 26223 raft_consensus.cc:479] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.025282 26223 leader_election.cc:243] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:01.025702 26223 leader_election.cc:243] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.028297 25602 ts_tablet_manager.cc:1173] T 9160f06bedfd4b57bba18f7eab8d7e1f P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.028741 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 69 ms (attempt = 2)
W0619 00:12:01.029036 25071 catalog_manager.cc:2910] Async tablet task 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 41e052fdd92b4c3d9803fa43433e60d5 consensus state has no leader
I0619 00:12:01.029201 25463 ts_tablet_manager.cc:1173] T 24281f894abb44419f18834dca2de65e P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.031885 25486 raft_consensus.cc:1081] T b34043f14f1c4edd97a1d4689ced5b32 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Refusing update from remote peer c007a00e65ad41a79da1a0c1050c55b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:01.033381 25463 tablet_service.cc:759] Processing CreateTablet for tablet 81f562d9bb3c4213b3970e0d79416bf0 (table=default.table_18b6ad6c2b1142c4a307bb3a4d56f328 [id=e878f682a0404fa5a518e9e0d120d456]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.037647 25469 tablet_service.cc:759] Processing CreateTablet for tablet b4f30bbe544f4f1f84ace8aab6adb319 (table=default.table_9a15f182e63744ac950a33cf371b30b4 [id=19547e0cea314ab8b0231bd548caf15e]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.038867 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 9d44681acacf4f04b6fcfbd4c9a18e87 (table default.table_6ace57434b5243c7bd22bd60752a065d [id=47a2f8c4d1f748a69c2bb0416bdb87e4]) successfully deleted
I0619 00:12:01.039248 25068 catalog_manager.cc:2891] Scheduling retry of 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC with a delay of 1045 ms (attempt = 7)
W0619 00:12:01.039530 25068 catalog_manager.cc:2910] Async tablet task 551fd06ba0c74bae9a152c7c2ab989f4 (table default.table_7d4d48d8304c46b5a13a0576fc34d663 [id=1cc3969f059d442eb594a9d4c719a8aa]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 551fd06ba0c74bae9a152c7c2ab989f4 consensus state has no leader
W0619 00:12:01.039429 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 3537c79c30a1401e8a78b387f08eff58 because the tablet was not found. No further retry: Not found: Tablet not found: 3537c79c30a1401e8a78b387f08eff58
W0619 00:12:01.040014 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a5c3203e9b6d423f810e1106369da3d7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a5c3203e9b6d423f810e1106369da3d7 already in progress: creating tablet
I0619 00:12:01.040264 25071 catalog_manager.cc:2891] Scheduling retry of a5c3203e9b6d423f810e1106369da3d7 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.040686 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a4c653a255984d3bac88df63362b9334 because the tablet was not found. No further retry: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
W0619 00:12:01.041329 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 0fee3be144c14edab6db58c99e3cda7d with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0fee3be144c14edab6db58c99e3cda7d already in progress: creating tablet
I0619 00:12:01.041563 25071 catalog_manager.cc:2891] Scheduling retry of 0fee3be144c14edab6db58c99e3cda7d Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 59 ms (attempt = 1)
W0619 00:12:01.041971 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet a4c653a255984d3bac88df63362b9334 because the tablet was not found. No further retry: Not found: Tablet not found: a4c653a255984d3bac88df63362b9334
W0619 00:12:01.042594 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet edfdedbd4eca4b62be24b1649f820d0e because the tablet was not found. No further retry: Not found: Tablet not found: edfdedbd4eca4b62be24b1649f820d0e
W0619 00:12:01.043035 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet 54adb588955b45b79de9752f254728ea because the tablet was not found. No further retry: Not found: Tablet not found: 54adb588955b45b79de9752f254728ea
I0619 00:12:01.034498 25558 leader_election.cc:379] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.044793 25600 ts_tablet_manager.cc:1173] T fbda888269cd4e25ba87671fcdeb2088 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.035316 25462 tablet_service.cc:759] Processing CreateTablet for tablet 740cd8ff815e4fd18466b0ca7fcb4b24 (table=default.table_45234f5ba11946f390d40924c779fe3e [id=12a7e07939ae4125b30d7473b37e1695]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.050096 25736 ts_tablet_manager.cc:1369] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:01.051167 25618 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb3c60fa2d8f4cafa12a5b0bc806433f" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:12:01.051625 25618 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.002s (vote granted)
I0619 00:12:01.051892 25623 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f155a418d0d547f7b775360f1a90ce01" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:12:01.052359 25623 raft_consensus.cc:2732] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 3.270s (vote granted)
I0619 00:12:01.052573 25623 raft_consensus.cc:2245] T f155a418d0d547f7b775360f1a90ce01 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:12:01.053688 25423 leader_election.cc:379] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:01.053997 25423 leader_election.cc:270] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:01.054468 26345 raft_consensus.cc:2732] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.979s (election complete)
I0619 00:12:01.054719 26345 raft_consensus.cc:2539] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:01.054994 26345 raft_consensus.cc:593] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:12:01.055403 26345 consensus_queue.cc:204] T 1cfd335c18c047fba2facc5194807abb P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.051877 25618 raft_consensus.cc:2245] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:12:01.057668 25464 tablet_service.cc:800] Processing DeleteTablet for tablet f155a418d0d547f7b775360f1a90ce01 with delete_type TABLET_DATA_DELETED (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.058578 25464 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.059237 26227 consensus_queue.cc:963] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:01.060647 25464 tablet_service.cc:800] Processing DeleteTablet for tablet 7277051d52dd4be1a6dfa06488c98293 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:00 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.052908 25467 tablet_service.cc:759] Processing CreateTablet for tablet 553a468dfae9489c8c11b784b55adef6 (table=default.table_4864844cc509452e8a846f7b0fc111cf [id=de883858812b45c493afaef046e18865]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.062381 25724 tablet_bootstrap.cc:427] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:01.062793 25724 ts_tablet_manager.cc:1045] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.120s	user 0.005s	sys 0.002s
I0619 00:12:01.064517 25464 tablet_service.cc:800] Processing DeleteTablet for tablet f155a418d0d547f7b775360f1a90ce01 with delete_type TABLET_DATA_DELETED (replaced by tablet 18dbdb33a25e487ab87ecb85af31fd55) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.064682 25724 raft_consensus.cc:325] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.065388 25724 raft_consensus.cc:351] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:01.065410 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 81f562d9bb3c4213b3970e0d79416bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.065619 25724 raft_consensus.cc:628] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:01.065871 25069 catalog_manager.cc:2891] Scheduling retry of 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC with a delay of 281 ms (attempt = 5)
W0619 00:12:01.066165 25069 catalog_manager.cc:2910] Async tablet task 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 24281f894abb44419f18834dca2de65e consensus state has no leader
I0619 00:12:01.066256 25724 consensus_queue.cc:227] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.068702 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1c4c69294cb945938004e5b9a0550e72" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8"
I0619 00:12:01.069141 25486 raft_consensus.cc:2784] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Advancing to term 2
I0619 00:12:01.069373 25486 raft_consensus.cc:2732] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.980s (vote granted)
I0619 00:12:01.069910 25469 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b4f30bbe544f4f1f84ace8aab6adb319. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.070729 25462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 740cd8ff815e4fd18466b0ca7fcb4b24. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.072708 25488 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cb3c60fa2d8f4cafa12a5b0bc806433f" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:01.074124 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 553a468dfae9489c8c11b784b55adef6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.078161 25086 catalog_manager.cc:3663] Got report from deleted tablet 2b82f9befb9144a7b4be080ac6d92a73 (table default.table_3e0e0119254a4be881b82ecf36174935 [id=a4ae0343c119473d90e61fab1e6df662]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:01.078889 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:12:01.082017 25419 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.1:32951, user_credentials={real_user=jenkins}} blocked reactor thread for 261424us
I0619 00:12:01.073154 25488 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.620s (vote granted)
I0619 00:12:01.083632 25488 raft_consensus.cc:2245] T cb3c60fa2d8f4cafa12a5b0bc806433f P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 0.
I0619 00:12:01.083746 25086 catalog_manager.cc:3663] Got report from deleted tablet 2b82f9befb9144a7b4be080ac6d92a73 (table default.table_3e0e0119254a4be881b82ecf36174935 [id=a4ae0343c119473d90e61fab1e6df662]) (Table deleted at 2018-06-19 00:11:54 PDT): Sending delete request for this tablet
I0619 00:12:01.084373 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:01.086652 25068 catalog_manager.cc:2891] Scheduling retry of a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC with a delay of 539 ms (attempt = 6)
W0619 00:12:01.086906 25068 catalog_manager.cc:2910] Async tablet task a4c3d22a3237408387518e3061470d01 (table default.table_12960b64943d44eda1316d619b74b8eb [id=18a0f5bc9de549e6965f46e858ce817a]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a4c3d22a3237408387518e3061470d01 consensus state has no leader
I0619 00:12:01.087030 25724 ts_tablet_manager.cc:1073] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.000s
I0619 00:12:01.041139 25730 ts_tablet_manager.cc:1382] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:01.088991 25730 log.cc:986] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/756057a7c34a4c61a261deaf849e433a
I0619 00:12:01.089570 25730 ts_tablet_manager.cc:1403] T 756057a7c34a4c61a261deaf849e433a P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:12:01.091084 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 271) took 5006ms. Request Metrics: {"tablet-delete.queue_time_us":4750547}
I0619 00:12:01.091828 25730 tablet_replica.cc:261] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:12:01.091862 26227 raft_consensus.cc:457] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.092100 26227 raft_consensus.cc:2732] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.535s (starting election)
I0619 00:12:01.092190 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:12:01.092473 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:12:01.092733 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:12:01.092977 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:12:01.093298 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:12:01.093545 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:12:01.093794 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(17e165e0995a4c029aa31392b9bbc04a)
I0619 00:12:01.094009 25730 raft_consensus.cc:2042] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 2 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:01.094303 25730 raft_consensus.cc:2069] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85 [term 2 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:01.096384 25730 ts_tablet_manager.cc:1369] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:01.097661 25486 raft_consensus.cc:2245] T 1c4c69294cb945938004e5b9a0550e72 P 10bda0b4856f4441b5aed24cdc4764a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395763cf325d4ce19859830a7ad35c85 in term 2.
I0619 00:12:01.092353 26227 raft_consensus.cc:479] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.098940 26227 leader_election.cc:243] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:01.099328 26227 leader_election.cc:243] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:01.098330 25736 ts_tablet_manager.cc:1382] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I0619 00:12:01.099952 25736 log.cc:986] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-1/wal/wals/3722b203a516466489c74ef6013eacae
I0619 00:12:01.103816 25724 ts_tablet_manager.cc:1028] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:01.104463 25724 tablet_bootstrap.cc:427] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:01.107084 25724 tablet_bootstrap.cc:578] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:01.117821 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 1034 ms (attempt = 7)
W0619 00:12:01.118160 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:12:01.118866 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5c3203e9b6d423f810e1106369da3d7 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.119282 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0fee3be144c14edab6db58c99e3cda7d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet 6755b36ed905430e8724d43920c9c0d8)
I0619 00:12:01.106393 25725 tablet_bootstrap.cc:427] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:12:01.120841 25725 ts_tablet_manager.cc:1045] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.312s	user 0.006s	sys 0.006s
I0619 00:12:01.121186 25600 tablet_service.cc:759] Processing CreateTablet for tablet 7f7525ea25e84258970ce75b2da6a7f2 (table=default.table_b2aff78ac49c4833bbfccd2f1593204a [id=e70c1b1159ce40bcadc3c5285d9cec7d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.121935 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7f7525ea25e84258970ce75b2da6a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.121894 25603 tablet_service.cc:759] Processing CreateTablet for tablet ef28fb43b9444a898fa158a81fc9cd0d (table=default.table_ed223177099d4877b476a5ec777d1d71 [id=ac710686c95f4e48b1f7819bf4ba1080]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.122529 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ef28fb43b9444a898fa158a81fc9cd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.122184 25725 raft_consensus.cc:325] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.122817 25725 raft_consensus.cc:351] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:01.123003 25725 raft_consensus.cc:628] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:12:01.123414 25725 consensus_queue.cc:227] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.124469 25602 tablet_service.cc:759] Processing CreateTablet for tablet fe79d880f6ce40eaafa97c31d6247abd (table=default.table_f02c65a4cf4e4707a03b3e4a3c9f8c99 [id=405058ace5c64033acdbe3ad4ced33a3]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.125085 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fe79d880f6ce40eaafa97c31d6247abd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.125128 25725 ts_tablet_manager.cc:1073] T 8e3a356933e749b9a4ade476ec30f475 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I0619 00:12:01.129330 25736 ts_tablet_manager.cc:1403] T 3722b203a516466489c74ef6013eacae P 10bda0b4856f4441b5aed24cdc4764a8: Deleting consensus metadata
I0619 00:12:01.132140 25603 ts_tablet_manager.cc:1173] T ef28fb43b9444a898fa158a81fc9cd0d P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.132530 25086 catalog_manager.cc:3663] Got report from deleted tablet e60ecc51c92c461889b360dde1dc5cf2 (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e): Sending delete request for this tablet
I0619 00:12:01.065026 25423 leader_election.cc:379] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Vote granted by peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344)
I0619 00:12:01.133152 25423 leader_election.cc:270] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:01.133282 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:12:01.133605 26227 raft_consensus.cc:2732] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 2.057s (election complete)
I0619 00:12:01.133857 26227 raft_consensus.cc:2539] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:01.134593 26227 raft_consensus.cc:593] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Becoming Leader. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Running, Role: LEADER
I0619 00:12:01.135020 26227 consensus_queue.cc:204] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.112529 25462 ts_tablet_manager.cc:1173] T 740cd8ff815e4fd18466b0ca7fcb4b24 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.141237 25469 ts_tablet_manager.cc:1173] T b4f30bbe544f4f1f84ace8aab6adb319 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.142259 25467 ts_tablet_manager.cc:1173] T 553a468dfae9489c8c11b784b55adef6 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.142992 25725 ts_tablet_manager.cc:1028] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrapping tablet
I0619 00:12:01.143330 25725 tablet_bootstrap.cc:427] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8: Bootstrap starting.
I0619 00:12:01.144256 25600 ts_tablet_manager.cc:1173] T 7f7525ea25e84258970ce75b2da6a7f2 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:01.144662 25423 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:39344, user_credentials={real_user=jenkins}} blocked reactor thread for 79658.4us
I0619 00:12:01.145422 25725 tablet_bootstrap.cc:578] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8: No blocks or log segments found. Creating new log.
I0619 00:12:01.145786 25463 ts_tablet_manager.cc:1173] T 81f562d9bb3c4213b3970e0d79416bf0 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.146190 25602 ts_tablet_manager.cc:1173] T fe79d880f6ce40eaafa97c31d6247abd P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.150574 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 406) took 3977ms. Request Metrics: {"tablet-delete.queue_time_us":3812504}
I0619 00:12:01.155033 25616 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4feb648bb30247b49a4abd49da95ea7d" candidate_uuid: "10bda0b4856f4441b5aed24cdc4764a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395763cf325d4ce19859830a7ad35c85" is_pre_election: true
I0619 00:12:01.155472 25616 raft_consensus.cc:2732] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.393s (vote granted)
I0619 00:12:01.155519 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:01.155725 25616 raft_consensus.cc:2245] T 4feb648bb30247b49a4abd49da95ea7d P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10bda0b4856f4441b5aed24cdc4764a8 in term 0.
I0619 00:12:01.156195 26296 raft_consensus.cc:457] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.156440 26296 raft_consensus.cc:2732] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.978s (starting election)
I0619 00:12:01.156939 25724 tablet_bootstrap.cc:427] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:01.156697 26296 raft_consensus.cc:479] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.157168 25071 catalog_manager.cc:2891] Scheduling retry of 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC with a delay of 268 ms (attempt = 5)
I0619 00:12:01.157254 25724 ts_tablet_manager.cc:1045] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.053s	user 0.005s	sys 0.000s
W0619 00:12:01.157424 25071 catalog_manager.cc:2910] Async tablet task 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 894de151fa3b4a4fbc962a5cafbccd3a consensus state has no leader
I0619 00:12:01.157678 26296 leader_election.cc:243] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:01.157755 25071 catalog_manager.cc:2891] Scheduling retry of beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC with a delay of 4132 ms (attempt = 9)
W0619 00:12:01.157975 25071 catalog_manager.cc:2910] Async tablet task beddf1523a854f5b9292c0e4013d7436 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet beddf1523a854f5b9292c0e4013d7436 consensus state has no leader
I0619 00:12:01.158082 26296 leader_election.cc:243] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.158267 25071 catalog_manager.cc:2891] Scheduling retry of c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC with a delay of 1044 ms (attempt = 7)
W0619 00:12:01.158488 25071 catalog_manager.cc:2910] Async tablet task c2cf56dada3c41d4920a4091a5fdfded (table default.table_9d96ad27d1d746f9a220f0524cd3c473 [id=0cd55331c72b42bfb2583be0283a06ef]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet c2cf56dada3c41d4920a4091a5fdfded consensus state has no leader
I0619 00:12:01.158776 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 756057a7c34a4c61a261deaf849e433a (table default.table_252d88d7d98b495e80affd5dc6eeb739 [id=f8e9404177f44b2cba84c5146b780133]) successfully deleted
I0619 00:12:01.158519 25724 raft_consensus.cc:325] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.159484 25724 raft_consensus.cc:351] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:01.159692 25724 raft_consensus.cc:628] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:01.160388 25603 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:01.160109 25724 consensus_queue.cc:227] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.160718 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 2b82f9befb9144a7b4be080ac6d92a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:01.161545 25602 tablet_service.cc:759] Processing CreateTablet for tablet 9a530ae8f1ac460fbc767943c5046c8b (table=default.table_d37acd08be134f80aaa9fca2a259551b [id=5fb84d7149e94f6cb19d1c077a06bc56]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.162166 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9a530ae8f1ac460fbc767943c5046c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.162477 26010 ts_manager.cc:81] Registered new tserver with Master: 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.170840 25558 leader_election.cc:379] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.171113 25558 leader_election.cc:270] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:01.171767 26341 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.427s (election complete)
I0619 00:12:01.171994 26341 raft_consensus.cc:2539] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:01.172180 26341 raft_consensus.cc:457] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:01.153357 25463 tablet_service.cc:759] Processing CreateTablet for tablet f2f581e6ef854e14800111a598987abd (table=default.table_b9cda64c37654d10b31d098b9a5e36f2 [id=3b25beee28d4405da5eced08eb9431ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.173162 25463 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f2f581e6ef854e14800111a598987abd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.175173 25558 leader_election.cc:379] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 2 election: Vote granted by peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.176874 25467 tablet_service.cc:759] Processing CreateTablet for tablet 558473fa04c4429289371ce2ae643421 (table=default.table_32abea053929403b9fd5b7a0b6189479 [id=479771692d41462cad49460cd7051e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.177453 25467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 558473fa04c4429289371ce2ae643421. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:01.178300 25423 leader_election.cc:306] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Not found: Tablet not found: 93930ffe285d48d080a8ca4793e871d9
I0619 00:12:01.172356 26341 raft_consensus.cc:2732] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.622s (starting election)
I0619 00:12:01.179291 25724 ts_tablet_manager.cc:1073] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.000s
I0619 00:12:01.179744 25724 ts_tablet_manager.cc:1028] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:01.179291 26341 raft_consensus.cc:2784] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:01.180079 25724 tablet_bootstrap.cc:427] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:01.184451 25730 ts_tablet_manager.cc:1382] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I0619 00:12:01.185612 25730 log.cc:986] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/17e165e0995a4c029aa31392b9bbc04a
I0619 00:12:01.151306 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 409) took 3947ms. Request Metrics: {"tablet-delete.queue_time_us":3940890}
I0619 00:12:01.191267 25423 leader_election.cc:270] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:01.191743 26227 raft_consensus.cc:2732] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8: Snoozing failure detection for 1.856s (election complete)
I0619 00:12:01.192059 26227 raft_consensus.cc:2487] T 93930ffe285d48d080a8ca4793e871d9 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:01.193773 25071 catalog_manager.cc:3091] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): tablet 3722b203a516466489c74ef6013eacae (table default.table_ec6a5ef017d74c7eb1777f1bf058a3da [id=a17737f9d2924cc09f372527fb91188b]) successfully deleted
I0619 00:12:01.196444 25469 tablet_service.cc:800] Processing DeleteTablet for tablet e60ecc51c92c461889b360dde1dc5cf2 with delete_type TABLET_DATA_DELETED (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.197150 25730 ts_tablet_manager.cc:1403] T 17e165e0995a4c029aa31392b9bbc04a P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:12:01.200073 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 272) took 5091ms. Request Metrics: {"tablet-delete.queue_time_us":4966342}
I0619 00:12:01.189471 26341 raft_consensus.cc:479] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.200947 26341 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:01.201365 26341 leader_election.cc:243] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: Requesting vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.192770 25724 tablet_bootstrap.cc:578] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:01.191018 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 410) took 3984ms. Request Metrics: {"tablet-delete.queue_time_us":3936451}
I0619 00:12:01.204995 25086 catalog_manager.cc:3663] Got report from deleted tablet 4feb648bb30247b49a4abd49da95ea7d (table default.table_9a8c9964bcff4effb2b624ef5bfdef1c [id=b9b59eba60624c4bbde989c6c7872e7f]) (Table deleted at 2018-06-19 00:11:57 PDT): Sending delete request for this tablet
I0619 00:12:01.205488 25086 catalog_manager.cc:3663] Got report from deleted tablet 1cfd335c18c047fba2facc5194807abb (table default.table_2af4c8dc619643729ba189a0dd8cd15f [id=ad26a268092e4cc4a44e6845dc511880]) (Table deleted at 2018-06-19 00:11:58 PDT): Sending delete request for this tablet
I0619 00:12:01.206912 25067 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 559 ms (attempt = 6)
W0619 00:12:01.207175 25067 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:01.214468 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "28f9fefa25d14aa089c34c9a0825946e" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:01.215934 25086 catalog_manager.cc:4229] T 00000000000000000000000000000000 P cb25d1bba5bf4c64914f300f122ec898: default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5] alter complete (version 1)
I0619 00:12:01.215937 25469 tablet_service.cc:800] Processing DeleteTablet for tablet fc7686daae274e0dba87703fe6924805 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:54 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.216459 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4feb648bb30247b49a4abd49da95ea7d on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:12:01.216840 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1cfd335c18c047fba2facc5194807abb on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.217525 25469 tablet_service.cc:800] Processing DeleteTablet for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.217829 25616 raft_consensus.cc:1081] T 41e052fdd92b4c3d9803fa43433e60d5 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Refusing update from remote peer 10bda0b4856f4441b5aed24cdc4764a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:01.220381 25725 tablet_bootstrap.cc:427] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8: No bootstrap required, opened a new log
I0619 00:12:01.220809 25725 ts_tablet_manager.cc:1045] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent bootstrapping tablet: real 0.078s	user 0.005s	sys 0.007s
W0619 00:12:01.222020 25071 catalog_manager.cc:3083] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet f75b8a4f1fd845fbab6f83f59b7cfc58 already in progress: creating tablet
I0619 00:12:01.222306 25071 catalog_manager.cc:2891] Scheduling retry of f75b8a4f1fd845fbab6f83f59b7cfc58 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 75 ms (attempt = 2)
I0619 00:12:01.221727 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 412) took 3959ms. Request Metrics: {"tablet-delete.queue_time_us":3940493}
I0619 00:12:01.222432 25725 raft_consensus.cc:325] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.223868 25725 raft_consensus.cc:351] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:01.224087 25725 raft_consensus.cc:628] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10bda0b4856f4441b5aed24cdc4764a8, State: Initialized, Role: FOLLOWER
I0619 00:12:01.224541 25725 consensus_queue.cc:227] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.226802 25486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "14179adec4b241be87f42625f5edefc6" candidate_uuid: "395763cf325d4ce19859830a7ad35c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10bda0b4856f4441b5aed24cdc4764a8" is_pre_election: true
I0619 00:12:01.223301 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 413) took 3960ms. Request Metrics: {"tablet-delete.queue_time_us":3959240}
W0619 00:12:01.228845 25558 leader_election.cc:306] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:01.227784 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 414) took 3922ms. Request Metrics: {"tablet-delete.queue_time_us":3900202}
I0619 00:12:01.229661 25469 tablet_service.cc:800] Processing DeleteTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f with delete_type TABLET_DATA_DELETED (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.229979 25602 ts_tablet_manager.cc:1173] T 9a530ae8f1ac460fbc767943c5046c8b P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:01.230206 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet d6c3c7f783cb43d59b46502261aeced3 because the tablet was not found. No further retry: Not found: Tablet not found: d6c3c7f783cb43d59b46502261aeced3
I0619 00:12:01.230065 25736 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.2:51757 (request call id 415) took 3923ms. Request Metrics: {"spinlock_wait_cycles":7552,"tablet-delete.queue_time_us":3901917}
W0619 00:12:01.230996 25071 catalog_manager.cc:3071] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): delete failed for tablet d6c3c7f783cb43d59b46502261aeced3 because the tablet was not found. No further retry: Not found: Tablet not found: d6c3c7f783cb43d59b46502261aeced3
I0619 00:12:01.231324 25736 tablet_replica.cc:261] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: stopping tablet replica
I0619 00:12:01.232697 25467 ts_tablet_manager.cc:1173] T 558473fa04c4429289371ce2ae643421 P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.232813 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 2065 ms (attempt = 8)
W0619 00:12:01.233068 25071 catalog_manager.cc:2910] Async tablet task 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 29ee1c11105e4d1e9ac74a247cd2953d consensus state has no leader
I0619 00:12:01.237051 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 273) took 5111ms. Request Metrics: {"tablet-delete.queue_time_us":5109944}
I0619 00:12:01.232988 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op CompactRowSetsOp(e0070a115e8540cda5f999394deac616)
I0619 00:12:01.237849 25730 tablet_replica.cc:261] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:12:01.238097 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(1dd3f7afe05f49199b18220f9c11ccd2)
I0619 00:12:01.238312 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(1dd3f7afe05f49199b18220f9c11ccd2)
I0619 00:12:01.238515 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(1dd3f7afe05f49199b18220f9c11ccd2)
I0619 00:12:01.238747 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(1dd3f7afe05f49199b18220f9c11ccd2)
I0619 00:12:01.238997 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(1dd3f7afe05f49199b18220f9c11ccd2)
I0619 00:12:01.239219 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(1dd3f7afe05f49199b18220f9c11ccd2)
I0619 00:12:01.239310 25463 ts_tablet_manager.cc:1173] T f2f581e6ef854e14800111a598987abd P 10bda0b4856f4441b5aed24cdc4764a8: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.239435 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(1dd3f7afe05f49199b18220f9c11ccd2)
I0619 00:12:01.239614 25730 raft_consensus.cc:2042] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:01.240080 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 17e165e0995a4c029aa31392b9bbc04a (table default.table_b1931959ceee4a6caa4a431f2c8234ba [id=84e03fd8999c4c67bfabc524e545d766]) successfully deleted
I0619 00:12:01.240783 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MinorDeltaCompactionOp(e0070a115e8540cda5f999394deac616)
I0619 00:12:01.241070 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op MajorDeltaCompactionOp(e0070a115e8540cda5f999394deac616)
I0619 00:12:01.241318 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op UndoDeltaBlockGCOp(e0070a115e8540cda5f999394deac616)
I0619 00:12:01.241590 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushMRSOp(e0070a115e8540cda5f999394deac616)
I0619 00:12:01.241861 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op FlushDeltaMemStoresOp(e0070a115e8540cda5f999394deac616)
I0619 00:12:01.242100 25736 maintenance_manager.cc:237] P 10bda0b4856f4441b5aed24cdc4764a8: Unregistered op LogGCOp(e0070a115e8540cda5f999394deac616)
I0619 00:12:01.242310 25736 raft_consensus.cc:2042] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:01.242524 25736 consensus_peers.cc:456] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): Closing peer: 395763cf325d4ce19859830a7ad35c85
I0619 00:12:01.242763 25736 consensus_peers.cc:456] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:12:01.243150 25736 raft_consensus.cc:2069] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:01.243892 25463 tablet_service.cc:800] Processing DeleteTablet for tablet cb3c60fa2d8f4cafa12a5b0bc806433f with delete_type TABLET_DATA_DELETED (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6) from {username='jenkins'} at 127.24.119.2:51757
I0619 00:12:01.239179 26363 tablet.cc:1200] T 71510e2112fd4dc9ba08de08648d7ad7 P 395763cf325d4ce19859830a7ad35c85: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:01.247076 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]). Expected version 1 got 0
I0619 00:12:01.239847 25730 consensus_peers.cc:456] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Closing peer: c007a00e65ad41a79da1a0c1050c55b4
I0619 00:12:01.248421 25730 consensus_peers.cc:456] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 -> Peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Closing peer: 10bda0b4856f4441b5aed24cdc4764a8
W0619 00:12:01.240591 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 0a33940e5347489993ab4679e9aeafde because the tablet was not found. No further retry: Not found: Tablet not found: 0a33940e5347489993ab4679e9aeafde
I0619 00:12:01.250561 25724 tablet_bootstrap.cc:427] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:01.250939 25724 ts_tablet_manager.cc:1045] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.071s	user 0.002s	sys 0.003s
I0619 00:12:01.257241 25736 ts_tablet_manager.cc:1369] T e0070a115e8540cda5f999394deac616 P 10bda0b4856f4441b5aed24cdc4764a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:01.259114 25725 ts_tablet_manager.cc:1073] T f75b8a4f1fd845fbab6f83f59b7cfc58 P 10bda0b4856f4441b5aed24cdc4764a8: Time spent starting tablet: real 0.038s	user 0.002s	sys 0.000s
W0619 00:12:01.259876 25067 connection.cc:512] server connection from 127.24.119.1:38914 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.259901 25553 connection.cc:512] client connection to 127.24.119.1:32951 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.260056 25555 connection.cc:512] server connection from 127.24.119.1:50230 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:50230): Connection reset by peer (error 104)
W0619 00:12:01.260413 25067 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0619 00:12:01.260351 25419 connection.cc:512] client connection to 127.24.119.1:32951 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.260519 25962 connection.cc:512] server connection from 127.24.119.1:54954 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:54954): Connection reset by peer (error 104)
W0619 00:12:01.260663 25419 consensus_peers.cc:434] T c244dec62a5b4ee492a9a110b656a561 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet c244dec62a5b4ee492a9a110b656a561. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:01.260762 25067 connection.cc:512] client connection to 127.24.119.1:32951 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.260929 25419 consensus_peers.cc:434] T 71510e2112fd4dc9ba08de08648d7ad7 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 71510e2112fd4dc9ba08de08648d7ad7. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:01.260977 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet 9a530ae8f1ac460fbc767943c5046c8b: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.261188 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 9a530ae8f1ac460fbc767943c5046c8b on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 27 ms (attempt = 1)
W0619 00:12:01.261178 25419 consensus_peers.cc:434] T 0a7470bfeab041f9895e491231b57eb7 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 0a7470bfeab041f9895e491231b57eb7. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:01.261437 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet ef28fb43b9444a898fa158a81fc9cd0d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.261418 25419 leader_election.cc:299] T 4feb648bb30247b49a4abd49da95ea7d P 10bda0b4856f4441b5aed24cdc4764a8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
I0619 00:12:01.261659 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet ef28fb43b9444a898fa158a81fc9cd0d on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 44 ms (attempt = 1)
W0619 00:12:01.261873 25419 consensus_peers.cc:434] T 7277051d52dd4be1a6dfa06488c98293 P 10bda0b4856f4441b5aed24cdc4764a8 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 7277051d52dd4be1a6dfa06488c98293. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:01.261871 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet fe79d880f6ce40eaafa97c31d6247abd: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.261950 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 25403
I0619 00:12:01.262075 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet fe79d880f6ce40eaafa97c31d6247abd on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 40 ms (attempt = 1)
W0619 00:12:01.262274 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet 7f7525ea25e84258970ce75b2da6a7f2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.262466 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 7f7525ea25e84258970ce75b2da6a7f2 on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 33 ms (attempt = 1)
W0619 00:12:01.262480 25553 leader_election.cc:299] T cb3c60fa2d8f4cafa12a5b0bc806433f P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.262691 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet 9160f06bedfd4b57bba18f7eab8d7e1f: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.262814 25553 leader_election.cc:299] T 14179adec4b241be87f42625f5edefc6 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
I0619 00:12:01.262876 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 9160f06bedfd4b57bba18f7eab8d7e1f on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 32 ms (attempt = 1)
W0619 00:12:01.263080 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet fbda888269cd4e25ba87671fcdeb2088: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.263093 25553 consensus_peers.cc:434] T e60ecc51c92c461889b360dde1dc5cf2 P 395763cf325d4ce19859830a7ad35c85 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet e60ecc51c92c461889b360dde1dc5cf2. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:01.263265 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet fbda888269cd4e25ba87671fcdeb2088 on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 24 ms (attempt = 1)
W0619 00:12:01.263463 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e60ecc51c92c461889b360dde1dc5cf2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.263478 25553 leader_election.cc:299] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
I0619 00:12:01.263725 25553 leader_election.cc:270] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:12:01.263875 25197 connection.cc:512] server connection from 127.24.119.1:38032 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.264420 25197 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
I0619 00:12:01.265446 25067 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 25 ms (attempt = 1)
W0619 00:12:01.265719 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 8e3a356933e749b9a4ade476ec30f475: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.265954 25067 catalog_manager.cc:2891] Scheduling retry of 8e3a356933e749b9a4ade476ec30f475 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 48 ms (attempt = 2)
W0619 00:12:01.266197 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 3e7c4b61f5314e40ad71371155a50b05: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.266424 25067 catalog_manager.cc:2891] Scheduling retry of 3e7c4b61f5314e40ad71371155a50b05 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 75 ms (attempt = 2)
W0619 00:12:01.266693 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 14179adec4b241be87f42625f5edefc6: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.266923 25067 catalog_manager.cc:2891] Scheduling retry of 14179adec4b241be87f42625f5edefc6 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 25 ms (attempt = 1)
W0619 00:12:01.267160 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet ef1566aa89b24620af70f2322df0dd56: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.267387 25067 catalog_manager.cc:2891] Scheduling retry of ef1566aa89b24620af70f2322df0dd56 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 52 ms (attempt = 1)
W0619 00:12:01.267649 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fc7686daae274e0dba87703fe6924805: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.267877 25067 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 25 ms (attempt = 1)
W0619 00:12:01.268115 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7277051d52dd4be1a6dfa06488c98293: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.268339 25067 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 23 ms (attempt = 1)
W0619 00:12:01.268582 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.268916 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 51 ms (attempt = 1)
W0619 00:12:01.269153 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e60ecc51c92c461889b360dde1dc5cf2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.269374 25067 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 62 ms (attempt = 1)
W0619 00:12:01.269608 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet b34043f14f1c4edd97a1d4689ced5b32: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.269872 25067 catalog_manager.cc:2891] Scheduling retry of b34043f14f1c4edd97a1d4689ced5b32 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 46 ms (attempt = 1)
I0619 00:12:01.269903 26341 raft_consensus.cc:2732] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.533s (election complete)
I0619 00:12:01.270133 26341 raft_consensus.cc:2487] T 28f9fefa25d14aa089c34c9a0825946e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:12:01.270143 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 8ca3a0771b9e41eeb0d14d886f970159: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.270347 25553 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
I0619 00:12:01.270391 25067 catalog_manager.cc:2891] Scheduling retry of 8ca3a0771b9e41eeb0d14d886f970159 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 77 ms (attempt = 2)
W0619 00:12:01.270668 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet bae6d308dae54b0394a364d2988ec6b3: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.270896 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 18 ms (attempt = 1)
W0619 00:12:01.271136 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f1b566b2b58f4955aa25ae990a4a8a1a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.270860 25724 raft_consensus.cc:325] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.271363 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 33 ms (attempt = 1)
I0619 00:12:01.271509 25724 raft_consensus.cc:351] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:12:01.271692 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f1b566b2b58f4955aa25ae990a4a8a1a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.271720 25724 raft_consensus.cc:628] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:01.271946 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 32 ms (attempt = 1)
W0619 00:12:01.272184 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet b34043f14f1c4edd97a1d4689ced5b32: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.272267 25069 catalog_manager.cc:2891] Scheduling retry of 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC with a delay of 558 ms (attempt = 6)
I0619 00:12:01.272411 25067 catalog_manager.cc:2891] Scheduling retry of b34043f14f1c4edd97a1d4689ced5b32 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 29 ms (attempt = 1)
W0619 00:12:01.272557 25069 catalog_manager.cc:2910] Async tablet task 28f9fefa25d14aa089c34c9a0825946e (table default.table_61eaea2a979142bbb875967b756eb1a1 [id=464afb6b12f4446b89a9682f1d407e8b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 28f9fefa25d14aa089c34c9a0825946e consensus state has no leader
I0619 00:12:01.272598 25724 consensus_queue.cc:227] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.273414 25730 raft_consensus.cc:2069] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:01.276660 26296 raft_consensus.cc:457] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.276885 26296 raft_consensus.cc:2732] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.505s (starting election)
I0619 00:12:01.277132 26296 raft_consensus.cc:479] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:12:01.277806 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet b34043f14f1c4edd97a1d4689ced5b32: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.278051 25067 catalog_manager.cc:2891] Scheduling retry of b34043f14f1c4edd97a1d4689ced5b32 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 58 ms (attempt = 1)
I0619 00:12:01.278106 26296 leader_election.cc:243] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:12:01.278319 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0ef73ff0abf44e5aa66cee31180c083c: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.278489 26296 leader_election.cc:243] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
I0619 00:12:01.278653 25067 catalog_manager.cc:2891] Scheduling retry of 0ef73ff0abf44e5aa66cee31180c083c Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 56 ms (attempt = 2)
W0619 00:12:01.278934 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 2a0b025496ee4087acadc27bbb6d4983: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.279180 25067 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 39 ms (attempt = 1)
W0619 00:12:01.279445 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f2d15bd8d6ce460c8422625ba060c6e0: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.279713 25067 catalog_manager.cc:2891] Scheduling retry of f2d15bd8d6ce460c8422625ba060c6e0 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 22 ms (attempt = 1)
W0619 00:12:01.279978 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e5b3be4f05474541b222b0c42884ec80: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.280133 25724 ts_tablet_manager.cc:1073] T dba1210d9c22454f8c9e0b0293583f0c P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.029s	user 0.003s	sys 0.001s
I0619 00:12:01.280220 25067 catalog_manager.cc:2891] Scheduling retry of e5b3be4f05474541b222b0c42884ec80 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 26 ms (attempt = 1)
I0619 00:12:01.280509 25724 ts_tablet_manager.cc:1028] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
W0619 00:12:01.280504 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fc7686daae274e0dba87703fe6924805: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.280814 25067 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 60 ms (attempt = 1)
I0619 00:12:01.280869 25724 tablet_bootstrap.cc:427] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:12:01.281100 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.281347 25067 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 19 ms (attempt = 1)
W0619 00:12:01.289742 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.290004 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 22 ms (attempt = 1)
W0619 00:12:01.290262 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fc7686daae274e0dba87703fe6924805: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.290411 25724 tablet_bootstrap.cc:578] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:01.290501 25067 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 29 ms (attempt = 1)
W0619 00:12:01.290812 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9d44681acacf4f04b6fcfbd4c9a18e87: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.291111 25067 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 33 ms (attempt = 1)
W0619 00:12:01.291369 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.291607 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 50 ms (attempt = 1)
W0619 00:12:01.291885 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.292129 25067 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 55 ms (attempt = 1)
W0619 00:12:01.292490 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet d3c7114381c0423982b9ccbff120c2a2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.292757 25067 catalog_manager.cc:2891] Scheduling retry of d3c7114381c0423982b9ccbff120c2a2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 50 ms (attempt = 1)
W0619 00:12:01.293007 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fe93c5dd62d94bdda099b9131482c033: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.293244 25067 catalog_manager.cc:2891] Scheduling retry of fe93c5dd62d94bdda099b9131482c033 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 18 ms (attempt = 1)
W0619 00:12:01.293506 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1c4c69294cb945938004e5b9a0550e72: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.293772 25067 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 65 ms (attempt = 1)
W0619 00:12:01.294028 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f1b566b2b58f4955aa25ae990a4a8a1a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.294267 25067 catalog_manager.cc:2891] Scheduling retry of f1b566b2b58f4955aa25ae990a4a8a1a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 75 ms (attempt = 2)
W0619 00:12:01.294513 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet d0624684b6b04ba3be260b2705dc00e1: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.294777 25067 catalog_manager.cc:2891] Scheduling retry of d0624684b6b04ba3be260b2705dc00e1 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 50 ms (attempt = 2)
W0619 00:12:01.295032 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 92fc5418174246bba92169339f264d62: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.295272 25067 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.295536 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 18dbdb33a25e487ab87ecb85af31fd55: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.295804 25067 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
W0619 00:12:01.296054 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet b8bc8d4a970843258ab5e9530f425d0f: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.296294 25067 catalog_manager.cc:2891] Scheduling retry of b8bc8d4a970843258ab5e9530f425d0f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 37 ms (attempt = 2)
W0619 00:12:01.296551 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 2b82f9befb9144a7b4be080ac6d92a73: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.296819 25067 catalog_manager.cc:2891] Scheduling retry of 2b82f9befb9144a7b4be080ac6d92a73 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 36 ms (attempt = 2)
W0619 00:12:01.297075 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.297320 25067 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 29 ms (attempt = 1)
I0619 00:12:01.297809 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f75b8a4f1fd845fbab6f83f59b7cfc58 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (Table deleted at 2018-06-19 00:11:53 PDT)
W0619 00:12:01.297576 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.298727 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.299000 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fb8e94875a8347a0a473c95a77c38795: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.299232 25067 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 18 ms (attempt = 1)
W0619 00:12:01.299494 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet b34043f14f1c4edd97a1d4689ced5b32: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.300563 25730 ts_tablet_manager.cc:1369] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:01.299623 25086 catalog_manager.cc:3663] Got report from deleted tablet 8e3a356933e749b9a4ade476ec30f475 (table default.table_b75186c4eb524b428a93fa8d4af3f24e [id=0d90b232db8e48ed9bfab0c59c6fbdba]) (replaced by tablet beddf1523a854f5b9292c0e4013d7436): Sending delete request for this tablet
I0619 00:12:01.301086 25086 catalog_manager.cc:3786] T 29ee1c11105e4d1e9ac74a247cd2953d P 10bda0b4856f4441b5aed24cdc4764a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "10bda0b4856f4441b5aed24cdc4764a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:01.302130 25086 catalog_manager.cc:3881] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) does not have the latest schema for tablet 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]). Expected version 1 got 0
I0619 00:12:01.303339 25067 catalog_manager.cc:2891] Scheduling retry of b34043f14f1c4edd97a1d4689ced5b32 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 76 ms (attempt = 2)
W0619 00:12:01.303642 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fc7686daae274e0dba87703fe6924805: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.303900 25067 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 69 ms (attempt = 2)
W0619 00:12:01.304148 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0a7470bfeab041f9895e491231b57eb7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.304414 25553 leader_election.cc:299] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.304711 25067 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
I0619 00:12:01.304671 25602 tablet_service.cc:759] Processing CreateTablet for tablet c000d4d9fb4141c89194288047917b83 (table=default.table_9932f993230644f484e23f0394689f84 [id=4a46530819ba4f7bbe7a535b90e161e3]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:01.304811 25553 consensus_peers.cc:434] T fc534f75faf345f487abeda223ad88db P 395763cf325d4ce19859830a7ad35c85 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet fc534f75faf345f487abeda223ad88db. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:01.305302 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c000d4d9fb4141c89194288047917b83. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:01.305243 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0a7470bfeab041f9895e491231b57eb7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.306658 25067 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 27 ms (attempt = 1)
W0619 00:12:01.306915 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 48eb7be170a64eb1bd869eba7067843d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.307157 25067 catalog_manager.cc:2891] Scheduling retry of 48eb7be170a64eb1bd869eba7067843d Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 22 ms (attempt = 1)
W0619 00:12:01.307415 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.307677 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 17 ms (attempt = 1)
W0619 00:12:01.307929 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9d44681acacf4f04b6fcfbd4c9a18e87: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.308166 25067 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 42 ms (attempt = 1)
W0619 00:12:01.308415 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 992f255e14544af3b8a89cd295944f29: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.308765 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e0070a115e8540cda5f999394deac616: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.309968 25067 catalog_manager.cc:2891] Scheduling retry of e0070a115e8540cda5f999394deac616 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 64 ms (attempt = 1)
W0619 00:12:01.310231 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0a7470bfeab041f9895e491231b57eb7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.310475 25067 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
W0619 00:12:01.310755 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1b40a55533f74f5eb90497d3c1064034: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.310997 25067 catalog_manager.cc:2891] Scheduling retry of 1b40a55533f74f5eb90497d3c1064034 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 16 ms (attempt = 1)
W0619 00:12:01.311259 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1cfd335c18c047fba2facc5194807abb: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.311764 25067 catalog_manager.cc:2891] Scheduling retry of 1cfd335c18c047fba2facc5194807abb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
W0619 00:12:01.312031 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0a7470bfeab041f9895e491231b57eb7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.316082 25724 tablet_bootstrap.cc:427] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:01.316465 25724 ts_tablet_manager.cc:1045] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.036s	user 0.005s	sys 0.001s
I0619 00:12:01.317540 25602 ts_tablet_manager.cc:1173] T c000d4d9fb4141c89194288047917b83 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.317919 25724 raft_consensus.cc:325] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.318799 25724 raft_consensus.cc:351] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:01.319003 25724 raft_consensus.cc:628] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:01.319509 25724 consensus_queue.cc:227] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.316160 25086 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
I0619 00:12:01.316942 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]). Expected version 2 got 0
I0619 00:12:01.327344 25087 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 55 ms (attempt = 1)
W0619 00:12:01.327606 25087 catalog_manager.cc:3945] Failed to send dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:12:01.328284 25067 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 25 ms (attempt = 1)
W0619 00:12:01.328558 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.328830 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 61 ms (attempt = 1)
W0619 00:12:01.329138 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet d6c3c7f783cb43d59b46502261aeced3: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.329387 25067 catalog_manager.cc:2891] Scheduling retry of d6c3c7f783cb43d59b46502261aeced3 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 27 ms (attempt = 1)
W0619 00:12:01.329670 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet d6c3c7f783cb43d59b46502261aeced3: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.329915 25067 catalog_manager.cc:2891] Scheduling retry of d6c3c7f783cb43d59b46502261aeced3 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 42 ms (attempt = 1)
W0619 00:12:01.330169 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e60ecc51c92c461889b360dde1dc5cf2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.330409 25067 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 62 ms (attempt = 2)
W0619 00:12:01.330687 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e0070a115e8540cda5f999394deac616: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.330930 25067 catalog_manager.cc:2891] Scheduling retry of e0070a115e8540cda5f999394deac616 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 18 ms (attempt = 1)
W0619 00:12:01.331183 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet ed3b0dace2e2475aa466039465b44a84: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.331423 25067 catalog_manager.cc:2891] Scheduling retry of ed3b0dace2e2475aa466039465b44a84 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 28 ms (attempt = 1)
W0619 00:12:01.331709 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.331951 25067 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 48 ms (attempt = 1)
I0619 00:12:01.332914 25087 catalog_manager.cc:3881] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) does not have the latest schema for tablet fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]). Expected version 2 got 0
I0619 00:12:01.333690 25087 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 18 ms (attempt = 1)
W0619 00:12:01.333950 25087 catalog_manager.cc:3945] Failed to send fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:12:01.334837 25600 tablet_service.cc:759] Processing CreateTablet for tablet a878d999b6ef4ffc83919d47d7c41a2c (table=default.table_272eb92753c343cdaca8e3d0f3f117b6 [id=b56c29fe825a4d849601b8e7c03ece74]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.335280 25602 tablet_service.cc:759] Processing CreateTablet for tablet 18a3861d32fa4c52975b7b3310c31448 (table=default.table_e04e900269934d1581d998ebe954a450 [id=0237925fe1984bc192919b479392cf9d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.335443 25600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a878d999b6ef4ffc83919d47d7c41a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.335988 25602 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 18a3861d32fa4c52975b7b3310c31448. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.336367 25724 ts_tablet_manager.cc:1073] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.000s
I0619 00:12:01.336740 25724 ts_tablet_manager.cc:1028] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:01.337065 25724 tablet_bootstrap.cc:427] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
I0619 00:12:01.337980 25603 tablet_service.cc:759] Processing CreateTablet for tablet a83dd81ebe6e4da38bed38df1ed48a39 (table=default.table_cb73054071fc44cf8f90838b548659d2 [id=52ecbfe13a394b1a9a9d5ce12aa5a93d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:01.338541 25603 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a83dd81ebe6e4da38bed38df1ed48a39. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:01.339287 25724 tablet_bootstrap.cc:578] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:01.348719 25069 catalog_manager.cc:2891] Scheduling retry of 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC with a delay of 515 ms (attempt = 6)
W0619 00:12:01.349011 25069 catalog_manager.cc:2910] Async tablet task 24281f894abb44419f18834dca2de65e (table default.table_7ead71299b534c358768a664635bb74d [id=388ad92a190b457497ffa5304c077739]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 24281f894abb44419f18834dca2de65e consensus state has no leader
W0619 00:12:01.350558 25553 consensus_peers.cc:434] T 1c4c69294cb945938004e5b9a0550e72 P 395763cf325d4ce19859830a7ad35c85 -> Peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Couldn't send request to peer c007a00e65ad41a79da1a0c1050c55b4 for tablet 1c4c69294cb945938004e5b9a0550e72. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:01.352052 25603 ts_tablet_manager.cc:1173] T a83dd81ebe6e4da38bed38df1ed48a39 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:01.352767 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0a7470bfeab041f9895e491231b57eb7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.353030 25067 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 62 ms (attempt = 1)
I0619 00:12:01.355163 25730 ts_tablet_manager.cc:1382] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:01.353226 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 43 ms (attempt = 2)
W0619 00:12:01.355492 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
W0619 00:12:01.355507 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet cb3c60fa2d8f4cafa12a5b0bc806433f: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.355813 25067 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 26 ms (attempt = 1)
W0619 00:12:01.356104 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet edfdedbd4eca4b62be24b1649f820d0e: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.356335 25067 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 16 ms (attempt = 1)
W0619 00:12:01.356575 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet edfdedbd4eca4b62be24b1649f820d0e: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.356830 25067 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
W0619 00:12:01.357066 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.357285 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 18 ms (attempt = 1)
W0619 00:12:01.357520 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e0070a115e8540cda5f999394deac616: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.357769 25067 catalog_manager.cc:2891] Scheduling retry of e0070a115e8540cda5f999394deac616 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 19 ms (attempt = 1)
W0619 00:12:01.358054 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 981d862d1bab4b14bb10e6fc2174e1cc: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.358283 25067 catalog_manager.cc:2891] Scheduling retry of 981d862d1bab4b14bb10e6fc2174e1cc Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 37 ms (attempt = 1)
W0619 00:12:01.358518 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.358772 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 56 ms (attempt = 1)
I0619 00:12:01.358804 25730 log.cc:986] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/1dd3f7afe05f49199b18220f9c11ccd2
W0619 00:12:01.359079 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 54adb588955b45b79de9752f254728ea: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.359335 25067 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 23 ms (attempt = 1)
I0619 00:12:01.359426 25730 ts_tablet_manager.cc:1403] T 1dd3f7afe05f49199b18220f9c11ccd2 P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
W0619 00:12:01.359652 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f24f9ab30ad04f39b5ac27e557606ab1: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.355257 25600 ts_tablet_manager.cc:1173] T a878d999b6ef4ffc83919d47d7c41a2c P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:01.359911 25067 catalog_manager.cc:2891] Scheduling retry of f24f9ab30ad04f39b5ac27e557606ab1 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 50 ms (attempt = 1)
W0619 00:12:01.360179 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e5051fc388d74ad39753c43b0ad25402: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.360422 25067 catalog_manager.cc:2891] Scheduling retry of e5051fc388d74ad39753c43b0ad25402 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 64 ms (attempt = 1)
I0619 00:12:01.361363 26341 raft_consensus.cc:457] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.361583 26341 raft_consensus.cc:2732] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.094s (starting election)
W0619 00:12:01.361685 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet b50379774e4d4fdfb823ab2dc919db30: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.361992 25067 catalog_manager.cc:2891] Scheduling retry of b50379774e4d4fdfb823ab2dc919db30 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 26 ms (attempt = 1)
I0619 00:12:01.361858 26341 raft_consensus.cc:479] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:12:01.362252 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet bda4e1ed3e8441ba90f63b78f67c61c5: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.362535 25067 catalog_manager.cc:2891] Scheduling retry of bda4e1ed3e8441ba90f63b78f67c61c5 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 27 ms (attempt = 1)
I0619 00:12:01.362891 26341 leader_election.cc:243] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:12:01.363179 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 4c92d341c4d94e4987524a056e703310: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.363281 26341 leader_election.cc:243] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
I0619 00:12:01.363515 25067 catalog_manager.cc:2891] Scheduling retry of 4c92d341c4d94e4987524a056e703310 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 64 ms (attempt = 1)
I0619 00:12:01.363896 25602 ts_tablet_manager.cc:1173] T 18a3861d32fa4c52975b7b3310c31448 P 395763cf325d4ce19859830a7ad35c85: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:01.364219 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 54adb588955b45b79de9752f254728ea: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.365509 25553 leader_election.cc:299] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.365847 25067 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 53 ms (attempt = 1)
W0619 00:12:01.366104 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet cbd67634f5704165b5bb948501697f0c: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.366338 25067 catalog_manager.cc:2891] Scheduling retry of cbd67634f5704165b5bb948501697f0c Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 29 ms (attempt = 1)
I0619 00:12:01.366762 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 274) took 5206ms. Request Metrics: {"mutex_wait_us":10224,"tablet-delete.queue_time_us":5077410}
W0619 00:12:01.366595 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 4feb648bb30247b49a4abd49da95ea7d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.367261 25067 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
I0619 00:12:01.367466 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 1dd3f7afe05f49199b18220f9c11ccd2 (table default.table_c24f1b02a5b54618aca5028342d84b7d [id=4b052b0654fd4377a294deee53dce6af]) successfully deleted
I0619 00:12:01.372025 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 276) took 5140ms. Request Metrics: {"tablet-delete.queue_time_us":5139596}
W0619 00:12:01.367524 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 2654c7e3655d46b0bac8251218b538e3: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.372630 25067 catalog_manager.cc:2891] Scheduling retry of 2654c7e3655d46b0bac8251218b538e3 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
W0619 00:12:01.372699 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2 because the tablet was not found. No further retry: Not found: Tablet not found: 1dd3f7afe05f49199b18220f9c11ccd2
W0619 00:12:01.372927 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet a284677c4b7f46e18e25edee6024e3eb: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.373196 25067 catalog_manager.cc:2891] Scheduling retry of a284677c4b7f46e18e25edee6024e3eb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 27 ms (attempt = 1)
W0619 00:12:01.373457 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet a284677c4b7f46e18e25edee6024e3eb: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.373724 25067 catalog_manager.cc:2891] Scheduling retry of a284677c4b7f46e18e25edee6024e3eb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 29 ms (attempt = 1)
W0619 00:12:01.373981 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e0070a115e8540cda5f999394deac616: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.374006 25724 tablet_bootstrap.cc:427] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:01.374230 25067 catalog_manager.cc:2891] Scheduling retry of e0070a115e8540cda5f999394deac616 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 40 ms (attempt = 1)
I0619 00:12:01.374395 25724 ts_tablet_manager.cc:1045] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.037s	user 0.005s	sys 0.001s
W0619 00:12:01.374583 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet c1223a8bb4fc4714b962ea5a49903675: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.374853 25067 catalog_manager.cc:2891] Scheduling retry of c1223a8bb4fc4714b962ea5a49903675 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 40 ms (attempt = 1)
W0619 00:12:01.375111 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet a4c653a255984d3bac88df63362b9334: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.375357 25067 catalog_manager.cc:2891] Scheduling retry of a4c653a255984d3bac88df63362b9334 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.375718 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0fee3be144c14edab6db58c99e3cda7d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.375962 25067 catalog_manager.cc:2891] Scheduling retry of 0fee3be144c14edab6db58c99e3cda7d Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
W0619 00:12:01.376220 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.376463 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 42 ms (attempt = 1)
I0619 00:12:01.376117 25724 raft_consensus.cc:325] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.376750 25724 raft_consensus.cc:351] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:12:01.376783 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 54adb588955b45b79de9752f254728ea: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.376946 25724 raft_consensus.cc:628] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:01.377063 25067 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
W0619 00:12:01.377355 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet a5c3203e9b6d423f810e1106369da3d7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.377601 25067 catalog_manager.cc:2891] Scheduling retry of a5c3203e9b6d423f810e1106369da3d7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 55 ms (attempt = 1)
W0619 00:12:01.377890 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet edfdedbd4eca4b62be24b1649f820d0e: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.377449 25724 consensus_queue.cc:227] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.373282 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 275) took 5141ms. Request Metrics: {"tablet-delete.queue_time_us":5125930}
I0619 00:12:01.378492 25067 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 92 ms (attempt = 3)
I0619 00:12:01.379041 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 280) took 5044ms. Request Metrics: {"tablet-delete.queue_time_us":5044059}
W0619 00:12:01.378793 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2 because the tablet was not found. No further retry: Not found: Tablet not found: 1dd3f7afe05f49199b18220f9c11ccd2
W0619 00:12:01.379675 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet eb36a22cf80841b58fd5ce9e6998bf00 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet eb36a22cf80841b58fd5ce9e6998bf00 already in progress: creating tablet
I0619 00:12:01.379890 25071 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 30 ms (attempt = 1)
W0619 00:12:01.380311 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e9a221a1d9814ef99a0b4ccaa04d1017: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.380560 25067 catalog_manager.cc:2891] Scheduling retry of e9a221a1d9814ef99a0b4ccaa04d1017 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 74 ms (attempt = 2)
W0619 00:12:01.386500 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0c1f35568fe24c6c8cd2b88871b4520d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.386780 25067 catalog_manager.cc:2891] Scheduling retry of 0c1f35568fe24c6c8cd2b88871b4520d Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 41 ms (attempt = 1)
W0619 00:12:01.387042 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet a4c653a255984d3bac88df63362b9334: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.389912 25730 tablet_replica.cc:261] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
I0619 00:12:01.390296 25724 ts_tablet_manager.cc:1073] T 9704177a036c469ab3d14211271cfd84 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
I0619 00:12:01.387286 25067 catalog_manager.cc:2891] Scheduling retry of a4c653a255984d3bac88df63362b9334 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
W0619 00:12:01.390743 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 075076aaa16e4ea8a596a7a8c52f103e: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.390915 25724 ts_tablet_manager.cc:1028] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:01.390990 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 18 ms (attempt = 1)
I0619 00:12:01.390552 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:01.391402 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(3e7724f967ae41d7b523055d1a4f58db)
W0619 00:12:01.391288 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 30a9d9fda8d34caeb8242c372439321b: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.391608 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:01.391696 25067 catalog_manager.cc:2891] Scheduling retry of 30a9d9fda8d34caeb8242c372439321b Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 1)
I0619 00:12:01.391837 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(3e7724f967ae41d7b523055d1a4f58db)
W0619 00:12:01.392024 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.392120 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:01.392303 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
I0619 00:12:01.392335 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(3e7724f967ae41d7b523055d1a4f58db)
I0619 00:12:01.392518 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(3e7724f967ae41d7b523055d1a4f58db)
W0619 00:12:01.392663 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.392724 25730 raft_consensus.cc:2042] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:01.396636 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
W0619 00:12:01.396904 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 54373917567a479d93dc8a3fd006d2fe: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.397145 25067 catalog_manager.cc:2891] Scheduling retry of 54373917567a479d93dc8a3fd006d2fe Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 46 ms (attempt = 1)
W0619 00:12:01.397392 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f155a418d0d547f7b775360f1a90ce01: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.397986 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.397815 26341 raft_consensus.cc:457] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:01.398290 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 41 ms (attempt = 1)
I0619 00:12:01.398296 26341 raft_consensus.cc:2732] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 2.142s (starting election)
W0619 00:12:01.398515 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.398758 25067 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 78 ms (attempt = 2)
I0619 00:12:01.398543 26341 raft_consensus.cc:479] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
W0619 00:12:01.398977 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 3e7724f967ae41d7b523055d1a4f58db: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.399510 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 74 ms (attempt = 3)
I0619 00:12:01.399605 26341 leader_election.cc:243] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951)
W0619 00:12:01.399799 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:12:01.399180 25067 catalog_manager.cc:2891] Scheduling retry of 3e7724f967ae41d7b523055d1a4f58db Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 56 ms (attempt = 1)
W0619 00:12:01.400173 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 3722b203a516466489c74ef6013eacae: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.400377 25067 catalog_manager.cc:2891] Scheduling retry of 3722b203a516466489c74ef6013eacae Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 60 ms (attempt = 1)
I0619 00:12:01.400449 26341 leader_election.cc:243] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512)
W0619 00:12:01.400598 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9d44681acacf4f04b6fcfbd4c9a18e87: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.400833 25067 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 16 ms (attempt = 1)
W0619 00:12:01.401053 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 54adb588955b45b79de9752f254728ea: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.401255 25067 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 39 ms (attempt = 1)
W0619 00:12:01.401471 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.401700 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.401918 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 86959e7dda374ba9ac23d3c1fd30d993: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.402217 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 86959e7dda374ba9ac23d3c1fd30d993: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.402755 25553 leader_election.cc:299] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.398080 25724 tablet_bootstrap.cc:427] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:12:01.402505 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 93930ffe285d48d080a8ca4793e871d9: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.405210 25067 catalog_manager.cc:2891] Scheduling retry of 93930ffe285d48d080a8ca4793e871d9 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 72 ms (attempt = 2)
I0619 00:12:01.405355 25724 tablet_bootstrap.cc:578] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
W0619 00:12:01.405473 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.405784 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 54 ms (attempt = 1)
I0619 00:12:01.407385 25730 raft_consensus.cc:2069] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:01.406075 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f75b8a4f1fd845fbab6f83f59b7cfc58: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.407784 25067 catalog_manager.cc:2891] Scheduling retry of f75b8a4f1fd845fbab6f83f59b7cfc58 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 78 ms (attempt = 2)
W0619 00:12:01.408044 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 86959e7dda374ba9ac23d3c1fd30d993: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.408352 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 86959e7dda374ba9ac23d3c1fd30d993: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.408694 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 6040e369e0fb445fb993d12dd883039b: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.408926 25067 catalog_manager.cc:2891] Scheduling retry of 6040e369e0fb445fb993d12dd883039b Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 51 ms (attempt = 2)
I0619 00:12:01.409025 25730 ts_tablet_manager.cc:1369] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:01.409418 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet d62715e230154e2abe126ef97feec7ce: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.410635 25071 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet eb36a22cf80841b58fd5ce9e6998bf00 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.411634 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 81 ms (attempt = 2)
I0619 00:12:01.411700 25602 tablet_service.cc:800] Processing DeleteTablet for tablet eb36a22cf80841b58fd5ce9e6998bf00 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:56 PDT) from {username='jenkins'} at 127.24.119.3:34285
W0619 00:12:01.411959 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.412226 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 28 ms (attempt = 1)
W0619 00:12:01.417786 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 3e7724f967ae41d7b523055d1a4f58db: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.420521 25730 ts_tablet_manager.cc:1382] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:01.420753 25067 catalog_manager.cc:2891] Scheduling retry of 3e7724f967ae41d7b523055d1a4f58db Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 59 ms (attempt = 1)
W0619 00:12:01.420991 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet d62715e230154e2abe126ef97feec7ce: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.421198 25067 catalog_manager.cc:2891] Scheduling retry of d62715e230154e2abe126ef97feec7ce Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 41 ms (attempt = 1)
W0619 00:12:01.421420 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 756057a7c34a4c61a261deaf849e433a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.421643 25067 catalog_manager.cc:2891] Scheduling retry of 756057a7c34a4c61a261deaf849e433a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 20 ms (attempt = 1)
W0619 00:12:01.421859 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0a33940e5347489993ab4679e9aeafde: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.422062 25067 catalog_manager.cc:2891] Scheduling retry of 0a33940e5347489993ab4679e9aeafde Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 64 ms (attempt = 1)
W0619 00:12:01.422276 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet eb36a22cf80841b58fd5ce9e6998bf00: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.422477 25067 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 17 ms (attempt = 1)
W0619 00:12:01.422709 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet a284677c4b7f46e18e25edee6024e3eb: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.422915 25067 catalog_manager.cc:2891] Scheduling retry of a284677c4b7f46e18e25edee6024e3eb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 42 ms (attempt = 2)
W0619 00:12:01.423125 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.423895 25730 log.cc:986] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_0.1529392303277196-25052/minicluster-data/ts-2/wal/wals/3e7724f967ae41d7b523055d1a4f58db
I0619 00:12:01.423322 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 53 ms (attempt = 1)
I0619 00:12:01.424886 25724 tablet_bootstrap.cc:427] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85: No bootstrap required, opened a new log
I0619 00:12:01.425274 25724 ts_tablet_manager.cc:1045] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85: Time spent bootstrapping tablet: real 0.034s	user 0.005s	sys 0.003s
W0619 00:12:01.425352 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.425597 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 48 ms (attempt = 1)
I0619 00:12:01.426767 25724 raft_consensus.cc:325] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.427392 25724 raft_consensus.cc:351] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:01.427593 25724 raft_consensus.cc:628] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395763cf325d4ce19859830a7ad35c85, State: Initialized, Role: FOLLOWER
I0619 00:12:01.427783 25071 catalog_manager.cc:2891] Scheduling retry of 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC with a delay of 536 ms (attempt = 6)
W0619 00:12:01.428050 25071 catalog_manager.cc:2910] Async tablet task 894de151fa3b4a4fbc962a5cafbccd3a (table default.table_4814ad0a42d246bda1bbfe321580b01f [id=9fbdae7ea5534755bcf89cf6a87f5748]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 894de151fa3b4a4fbc962a5cafbccd3a consensus state has no leader
I0619 00:12:01.428278 25724 consensus_queue.cc:227] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10bda0b4856f4441b5aed24cdc4764a8" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 38512 } } peers { permanent_uuid: "c007a00e65ad41a79da1a0c1050c55b4" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 32951 } } peers { permanent_uuid: "395763cf325d4ce19859830a7ad35c85" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 39344 } }
I0619 00:12:01.432209 25087 catalog_manager.cc:3663] Got report from deleted tablet 8ca3a0771b9e41eeb0d14d886f970159 (table default.table_53e73552aa6d4161a595d779a9fc96cc [id=1697152e5d9f409ab68219b93b01250f]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:12:01.432951 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:12:01.434939 25730 ts_tablet_manager.cc:1403] T 3e7724f967ae41d7b523055d1a4f58db P 395763cf325d4ce19859830a7ad35c85: Deleting consensus metadata
I0619 00:12:01.437372 25087 catalog_manager.cc:3663] Got report from deleted tablet 8ca3a0771b9e41eeb0d14d886f970159 (table default.table_53e73552aa6d4161a595d779a9fc96cc [id=1697152e5d9f409ab68219b93b01250f]) (Table deleted at 2018-06-19 00:11:55 PDT): Sending delete request for this tablet
I0619 00:12:01.438031 25087 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ca3a0771b9e41eeb0d14d886f970159 on 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:12:01.435022 25602 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:01.441072 25600 tablet_service.cc:800] Processing DeleteTablet for tablet 8ca3a0771b9e41eeb0d14d886f970159 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:11:55 PDT) from {username='jenkins'} at 127.24.119.3:34285
I0619 00:12:01.443207 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 281) took 5091ms. Request Metrics: {"mutex_wait_us":323,"tablet-delete.queue_time_us":5032623}
I0619 00:12:01.443949 25071 catalog_manager.cc:3091] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): tablet 3e7724f967ae41d7b523055d1a4f58db (table default.table_2e9154601fc34c21a4a6c3e101a644dc [id=6a6f8955a630493cba4ab20b54d5b27f]) successfully deleted
I0619 00:12:01.444608 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 282) took 5090ms. Request Metrics: {"tablet-delete.queue_time_us":5086809}
W0619 00:12:01.445706 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet d3c7114381c0423982b9ccbff120c2a2 because the tablet was not found. No further retry: Not found: Tablet not found: d3c7114381c0423982b9ccbff120c2a2
I0619 00:12:01.446902 25724 ts_tablet_manager.cc:1073] T 8ca3a0771b9e41eeb0d14d886f970159 P 395763cf325d4ce19859830a7ad35c85: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I0619 00:12:01.447299 25724 ts_tablet_manager.cc:1028] T 3e7c4b61f5314e40ad71371155a50b05 P 395763cf325d4ce19859830a7ad35c85: Bootstrapping tablet
I0619 00:12:01.447672 25724 tablet_bootstrap.cc:427] T 3e7c4b61f5314e40ad71371155a50b05 P 395763cf325d4ce19859830a7ad35c85: Bootstrap starting.
W0619 00:12:01.447707 25067 catalog_manager.cc:2910] Async tablet task bae6d308dae54b0394a364d2988ec6b3 (table default.table_f7cff01960d64d739ae50955fabe7254 [id=b0b4e2e4ca2f43a8aeb63ab79213c17c]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet bae6d308dae54b0394a364d2988ec6b3 consensus state has no leader
I0619 00:12:01.448150 25067 catalog_manager.cc:2891] Scheduling retry of 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC with a delay of 4125 ms (attempt = 9)
W0619 00:12:01.448423 25067 catalog_manager.cc:2910] Async tablet task 1b15ca9f6e4d4d32bbff1f041a5cb13c (table default.table_c74c58e16b0e4279ad767a97fe30c436 [id=191ee261cfbe4ca3b04553067de4e26d]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b15ca9f6e4d4d32bbff1f041a5cb13c consensus state has no leader
W0619 00:12:01.448835 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet a83dd81ebe6e4da38bed38df1ed48a39: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.449084 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet a83dd81ebe6e4da38bed38df1ed48a39 on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 25 ms (attempt = 1)
W0619 00:12:01.449343 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet a878d999b6ef4ffc83919d47d7c41a2c: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.449584 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet a878d999b6ef4ffc83919d47d7c41a2c on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 18 ms (attempt = 1)
W0619 00:12:01.449867 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet 18a3861d32fa4c52975b7b3310c31448: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.450109 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 18a3861d32fa4c52975b7b3310c31448 on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 49 ms (attempt = 1)
W0619 00:12:01.450369 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Create Tablet RPC failed for tablet c000d4d9fb4141c89194288047917b83: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.450631 25067 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet c000d4d9fb4141c89194288047917b83 on TS c007a00e65ad41a79da1a0c1050c55b4 with a delay of 52 ms (attempt = 1)
I0619 00:12:01.450098 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 284) took 5081ms. Request Metrics: {"tablet-delete.queue_time_us":5076885}
I0619 00:12:01.451164 25067 catalog_manager.cc:2891] Scheduling retry of a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC with a delay of 1030 ms (attempt = 7)
W0619 00:12:01.451236 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet 0c1f35568fe24c6c8cd2b88871b4520d with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 0c1f35568fe24c6c8cd2b88871b4520d already in progress: creating tablet
I0619 00:12:01.451464 25071 catalog_manager.cc:2891] Scheduling retry of 0c1f35568fe24c6c8cd2b88871b4520d Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 53 ms (attempt = 1)
W0619 00:12:01.451488 25067 catalog_manager.cc:2910] Async tablet task a7a3d73c048749b489660d109fa15158 (table default.table_71308ace98a941b8922f99345c2300a2 [id=f7e13f6492c0456e930628e95fd81358]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a7a3d73c048749b489660d109fa15158 consensus state has no leader
I0619 00:12:01.451892 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.452314 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1b40a55533f74f5eb90497d3c1064034 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.452517 25724 tablet_bootstrap.cc:578] T 3e7c4b61f5314e40ad71371155a50b05 P 395763cf325d4ce19859830a7ad35c85: No blocks or log segments found. Creating new log.
I0619 00:12:01.452914 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.453001 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 290) took 4835ms. Request Metrics: {"tablet-delete.queue_time_us":4828830}
I0619 00:12:01.454762 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet eb36a22cf80841b58fd5ce9e6998bf00 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:12:01.454916 25071 catalog_manager.cc:3071] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet b34043f14f1c4edd97a1d4689ced5b32 because the tablet was not found. No further retry: Not found: Tablet not found: b34043f14f1c4edd97a1d4689ced5b32
I0619 00:12:01.455216 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.455484 25069 catalog_manager.cc:2891] Scheduling retry of a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC with a delay of 4113 ms (attempt = 9)
W0619 00:12:01.455768 25069 catalog_manager.cc:2910] Async tablet task a858d8f5709648a3ba7649315ccc9662 (table default.table_c9352e571d1b4c3da9d5ef85ec9e9139 [id=2f4c8afb95b945138a65de871be4b5cf]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet a858d8f5709648a3ba7649315ccc9662 consensus state has no leader
I0619 00:12:01.455621 25730 rpcz_store.cc:265] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.24.119.3:34285 (request call id 294) took 4781ms. Request Metrics: {"tablet-delete.queue_time_us":4774110}
I0619 00:12:01.455663 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.456769 25730 tablet_replica.cc:261] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: stopping tablet replica
W0619 00:12:01.456341 25071 catalog_manager.cc:3083] TS 395763cf325d4ce19859830a7ad35c85 (127.24.119.3:39344): delete failed for tablet a5c3203e9b6d423f810e1106369da3d7 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a5c3203e9b6d423f810e1106369da3d7 already in progress: creating tablet
I0619 00:12:01.457010 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op CompactRowSetsOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:01.456992 25071 catalog_manager.cc:2891] Scheduling retry of a5c3203e9b6d423f810e1106369da3d7 Delete Tablet RPC for TS=395763cf325d4ce19859830a7ad35c85 with a delay of 50 ms (attempt = 1)
I0619 00:12:01.457214 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MinorDeltaCompactionOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:01.457602 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op MajorDeltaCompactionOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:01.457602 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 075076aaa16e4ea8a596a7a8c52f103e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet dba1210d9c22454f8c9e0b0293583f0c)
I0619 00:12:01.457839 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op UndoDeltaBlockGCOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:01.458070 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushMRSOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:01.458289 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op FlushDeltaMemStoresOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:01.458125 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bae6d308dae54b0394a364d2988ec6b3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:01.458482 25730 maintenance_manager.cc:237] P 395763cf325d4ce19859830a7ad35c85: Unregistered op LogGCOp(9d44681acacf4f04b6fcfbd4c9a18e87)
I0619 00:12:01.458659 25730 raft_consensus.cc:2042] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:01.458973 25730 raft_consensus.cc:2069] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:01.460520 25730 ts_tablet_manager.cc:1369] T 9d44681acacf4f04b6fcfbd4c9a18e87 P 395763cf325d4ce19859830a7ad35c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:01.458837 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fe93c5dd62d94bdda099b9131482c033 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:01.461091 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fb8e94875a8347a0a473c95a77c38795 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:01.461493 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.461920 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.462324 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:01.466637 25071 catalog_manager.cc:2891] Scheduling retry of 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC with a delay of 1060 ms (attempt = 7)
W0619 00:12:01.466652 25558 connection.cc:512] client connection to 127.24.119.2:38512 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.466930 25071 catalog_manager.cc:2910] Async tablet task 455d8e1ee5be484b8005312d2d886821 (table default.table_c0119de186df44faa67a07b4cd18cf17 [id=f7588ecc502740d1bb371211ac284938]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet 455d8e1ee5be484b8005312d2d886821 consensus state has no leader
W0619 00:12:01.466967 25558 leader_election.cc:299] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.467073 25198 connection.cc:512] server connection from 127.24.119.2:46639 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.2:46639): Connection reset by peer (error 104)
W0619 00:12:01.467185 25071 connection.cc:512] client connection to 127.24.119.2:38512 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.467217 25558 leader_election.cc:270] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:12:01.467397 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 41e052fdd92b4c3d9803fa43433e60d5: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.467605 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 8201 ms (attempt = 10)
W0619 00:12:01.467723 25890 connection.cc:512] server connection from 127.24.119.2:48241 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.2:48241): Connection reset by peer (error 104)
W0619 00:12:01.467836 25558 leader_election.cc:299] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.467911 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 41e052fdd92b4c3d9803fa43433e60d5: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.468082 25558 leader_election.cc:270] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:01.468129 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 2094 ms (attempt = 8)
W0619 00:12:01.468394 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 71510e2112fd4dc9ba08de08648d7ad7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.468497 26296 raft_consensus.cc:2732] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.975s (election complete)
I0619 00:12:01.468600 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 4126 ms (attempt = 9)
W0619 00:12:01.468632 25558 leader_election.cc:299] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
I0619 00:12:01.468744 26296 raft_consensus.cc:2487] T fa0ab4b6eef04ae08eb8994b49ca531e P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:12:01.468899 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet a83dd81ebe6e4da38bed38df1ed48a39: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.468962 25558 leader_election.cc:270] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:01.469019 26296 raft_consensus.cc:2732] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.538s (election complete)
I0619 00:12:01.469106 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet a83dd81ebe6e4da38bed38df1ed48a39 on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 25 ms (attempt = 1)
I0619 00:12:01.469241 26296 raft_consensus.cc:2487] T 2b82f9befb9144a7b4be080ac6d92a73 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:12:01.469332 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet a878d999b6ef4ffc83919d47d7c41a2c: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.469466 26296 raft_consensus.cc:2732] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85: Snoozing failure detection for 1.820s (election complete)
I0619 00:12:01.469537 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet a878d999b6ef4ffc83919d47d7c41a2c on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 35 ms (attempt = 1)
I0619 00:12:01.469681 26296 raft_consensus.cc:2487] T 8b437c976cfd41aa8f2f2f38b3762005 P 395763cf325d4ce19859830a7ad35c85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W0619 00:12:01.469780 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 29ee1c11105e4d1e9ac74a247cd2953d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.469885 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 25537
I0619 00:12:01.469983 25071 catalog_manager.cc:2891] Scheduling retry of 29ee1c11105e4d1e9ac74a247cd2953d (table default.table_a2078fb395364185b90d58ba4ed9318d [id=e064506301e54738aada2d76e8bab056]) Alter Table RPC with a delay of 41 ms (attempt = 1)
W0619 00:12:01.470885 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 8e3a356933e749b9a4ade476ec30f475: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.471045 25071 catalog_manager.cc:2891] Scheduling retry of 8e3a356933e749b9a4ade476ec30f475 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 17 ms (attempt = 1)
W0619 00:12:01.471205 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet 18a3861d32fa4c52975b7b3310c31448: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.471348 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 18a3861d32fa4c52975b7b3310c31448 on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 23 ms (attempt = 1)
W0619 00:12:01.471501 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet c000d4d9fb4141c89194288047917b83: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.471659 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet c000d4d9fb4141c89194288047917b83 on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 39 ms (attempt = 1)
W0619 00:12:01.471825 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet f75b8a4f1fd845fbab6f83f59b7cfc58: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.471963 25071 catalog_manager.cc:2891] Scheduling retry of f75b8a4f1fd845fbab6f83f59b7cfc58 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 84 ms (attempt = 3)
W0619 00:12:01.472113 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 71510e2112fd4dc9ba08de08648d7ad7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.472252 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 24 ms (attempt = 1)
W0619 00:12:01.472429 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 1cfd335c18c047fba2facc5194807abb: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.472569 25071 catalog_manager.cc:2891] Scheduling retry of 1cfd335c18c047fba2facc5194807abb Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 51 ms (attempt = 1)
W0619 00:12:01.472761 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 4feb648bb30247b49a4abd49da95ea7d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.472914 25071 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 30 ms (attempt = 1)
W0619 00:12:01.473068 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 0fee3be144c14edab6db58c99e3cda7d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.473206 25071 catalog_manager.cc:2891] Scheduling retry of 0fee3be144c14edab6db58c99e3cda7d Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 65 ms (attempt = 2)
W0619 00:12:01.473354 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet a5c3203e9b6d423f810e1106369da3d7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.473492 25071 catalog_manager.cc:2891] Scheduling retry of a5c3203e9b6d423f810e1106369da3d7 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 71 ms (attempt = 2)
W0619 00:12:01.473657 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 41e052fdd92b4c3d9803fa43433e60d5: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.473827 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 78 ms (attempt = 3)
W0619 00:12:01.474020 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet 9a530ae8f1ac460fbc767943c5046c8b: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.474210 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 9a530ae8f1ac460fbc767943c5046c8b on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 52 ms (attempt = 1)
I0619 00:12:01.474274 25068 catalog_manager.cc:2891] Scheduling retry of fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC with a delay of 130 ms (attempt = 4)
W0619 00:12:01.474433 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet ef28fb43b9444a898fa158a81fc9cd0d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.474540 25068 catalog_manager.cc:2910] Async tablet task fa0ab4b6eef04ae08eb8994b49ca531e (table default.table_aad8b0f2529e4f288b86d31a9769d357 [id=b9bfdc8e55474269827835656bfecf3b]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet fa0ab4b6eef04ae08eb8994b49ca531e consensus state has no leader
I0619 00:12:01.475136 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet ef28fb43b9444a898fa158a81fc9cd0d on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 19 ms (attempt = 1)
W0619 00:12:01.475543 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet fe79d880f6ce40eaafa97c31d6247abd: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.475716 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet fe79d880f6ce40eaafa97c31d6247abd on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 34 ms (attempt = 1)
W0619 00:12:01.475878 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 41e052fdd92b4c3d9803fa43433e60d5: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.476020 25071 catalog_manager.cc:2891] Scheduling retry of 41e052fdd92b4c3d9803fa43433e60d5 (table default.table_1139cb4bdb734efd950a2c4f0a1e4b95 [id=e3efb205417f4b40b4f48094be7d1e1e]) Alter Table RPC with a delay of 42 ms (attempt = 1)
W0619 00:12:01.476202 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet 7f7525ea25e84258970ce75b2da6a7f2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.476344 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 7f7525ea25e84258970ce75b2da6a7f2 on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 48 ms (attempt = 1)
W0619 00:12:01.476496 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 71510e2112fd4dc9ba08de08648d7ad7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.476660 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 2075 ms (attempt = 8)
W0619 00:12:01.476851 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet 9160f06bedfd4b57bba18f7eab8d7e1f: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.476997 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 9160f06bedfd4b57bba18f7eab8d7e1f on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 46 ms (attempt = 1)
W0619 00:12:01.477174 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet fbda888269cd4e25ba87671fcdeb2088: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.477315 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet fbda888269cd4e25ba87671fcdeb2088 on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 33 ms (attempt = 1)
W0619 00:12:01.477489 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 14179adec4b241be87f42625f5edefc6: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.477649 25071 catalog_manager.cc:2891] Scheduling retry of 14179adec4b241be87f42625f5edefc6 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 32 ms (attempt = 1)
W0619 00:12:01.477815 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet ef1566aa89b24620af70f2322df0dd56: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.477953 25071 catalog_manager.cc:2891] Scheduling retry of ef1566aa89b24620af70f2322df0dd56 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 53 ms (attempt = 1)
W0619 00:12:01.478104 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 71510e2112fd4dc9ba08de08648d7ad7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.478242 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 20 ms (attempt = 1)
W0619 00:12:01.478422 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 0a7470bfeab041f9895e491231b57eb7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.478564 25071 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 49 ms (attempt = 1)
W0619 00:12:01.478758 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet cb3c60fa2d8f4cafa12a5b0bc806433f: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.478901 25071 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 37 ms (attempt = 1)
W0619 00:12:01.479060 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet cb3c60fa2d8f4cafa12a5b0bc806433f: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.479209 25071 catalog_manager.cc:2891] Scheduling retry of cb3c60fa2d8f4cafa12a5b0bc806433f Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.479367 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 9d44681acacf4f04b6fcfbd4c9a18e87: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.479508 25071 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 60 ms (attempt = 1)
W0619 00:12:01.479684 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet fc7686daae274e0dba87703fe6924805: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.479825 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 33 ms (attempt = 1)
W0619 00:12:01.479984 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e60ecc51c92c461889b360dde1dc5cf2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.480126 25071 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 41 ms (attempt = 1)
W0619 00:12:01.480285 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet 558473fa04c4429289371ce2ae643421: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.480427 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 558473fa04c4429289371ce2ae643421 on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 21 ms (attempt = 1)
I0619 00:12:01.474952 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 4feb648bb30247b49a4abd49da95ea7d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:12:01.480844 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.481106 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0fee3be144c14edab6db58c99e3cda7d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 6755b36ed905430e8724d43920c9c0d8)
I0619 00:12:01.481361 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:12:01.481636 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:12:01.481901 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 756057a7c34a4c61a261deaf849e433a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.482152 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 18dbdb33a25e487ab87ecb85af31fd55 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:01.482403 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.482672 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1cfd335c18c047fba2facc5194807abb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.482936 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f2d15bd8d6ce460c8422625ba060c6e0 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 740cd8ff815e4fd18466b0ca7fcb4b24)
I0619 00:12:01.483191 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.483443 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 48eb7be170a64eb1bd869eba7067843d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.486042 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.486317 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7277051d52dd4be1a6dfa06488c98293 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:01.490089 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14179adec4b241be87f42625f5edefc6 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
I0619 00:12:01.490502 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:12:01.480588 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Create Tablet RPC failed for tablet f2f581e6ef854e14800111a598987abd: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.490955 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.491391 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e60ecc51c92c461889b360dde1dc5cf2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 28f9fefa25d14aa089c34c9a0825946e)
I0619 00:12:01.491809 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b50379774e4d4fdfb823ab2dc919db30 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 209b4d3f41df46de9d420e273fa1d1cb)
I0619 00:12:01.492211 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5b3be4f05474541b222b0c42884ec80 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:01.492601 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cb3c60fa2d8f4cafa12a5b0bc806433f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 8252076f40e1409681d92ba4cca0b0e6)
I0619 00:12:01.493023 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bda4e1ed3e8441ba90f63b78f67c61c5 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:12:01.493415 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:01.494096 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.494484 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:12:01.494891 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.495280 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ed3b0dace2e2475aa466039465b44a84 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.495667 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet cbd67634f5704165b5bb948501697f0c on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 507f7ed270af42ce8f404d55b5cae468)
I0619 00:12:01.496107 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:01.496498 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:01.496917 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fc7686daae274e0dba87703fe6924805 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:01.497309 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:01.497720 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:01.490972 25071 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet f2f581e6ef854e14800111a598987abd on TS 10bda0b4856f4441b5aed24cdc4764a8 with a delay of 54 ms (attempt = 1)
W0619 00:12:01.498196 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet f155a418d0d547f7b775360f1a90ce01: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.498417 25071 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 16 ms (attempt = 1)
I0619 00:12:01.498472 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f1b566b2b58f4955aa25ae990a4a8a1a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:12:01.498673 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 7277051d52dd4be1a6dfa06488c98293: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.498890 25071 catalog_manager.cc:2891] Scheduling retry of 7277051d52dd4be1a6dfa06488c98293 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 23 ms (attempt = 1)
I0619 00:12:01.499002 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:12:01.499114 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet fc7686daae274e0dba87703fe6924805: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.499326 25071 catalog_manager.cc:2891] Scheduling retry of fc7686daae274e0dba87703fe6924805 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 63 ms (attempt = 1)
W0619 00:12:01.499550 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet f155a418d0d547f7b775360f1a90ce01: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.499788 25071 catalog_manager.cc:2891] Scheduling retry of f155a418d0d547f7b775360f1a90ce01 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 39 ms (attempt = 1)
I0619 00:12:01.499867 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b82f9befb9144a7b4be080ac6d92a73 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
W0619 00:12:01.500015 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e60ecc51c92c461889b360dde1dc5cf2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.500227 25071 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 43 ms (attempt = 1)
I0619 00:12:01.500357 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
W0619 00:12:01.500458 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 7277051d52dd4be1a6dfa06488c98293: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.500802 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 71510e2112fd4dc9ba08de08648d7ad7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.500869 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b8bc8d4a970843258ab5e9530f425d0f on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:01.501019 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 1031 ms (attempt = 7)
W0619 00:12:01.501298 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet eb36a22cf80841b58fd5ce9e6998bf00: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.501384 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.501515 25071 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 41 ms (attempt = 2)
W0619 00:12:01.501767 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 0c1f35568fe24c6c8cd2b88871b4520d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.501899 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2a0b025496ee4087acadc27bbb6d4983 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:59 PDT)
I0619 00:12:01.501982 25071 catalog_manager.cc:2891] Scheduling retry of 0c1f35568fe24c6c8cd2b88871b4520d Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 75 ms (attempt = 2)
I0619 00:12:01.502380 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e0070a115e8540cda5f999394deac616 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:12:01.502606 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Alter Table RPC failed for tablet 71510e2112fd4dc9ba08de08648d7ad7: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.502848 25071 catalog_manager.cc:2891] Scheduling retry of 71510e2112fd4dc9ba08de08648d7ad7 (table default.table_c57b5014fe19429b8fde87e3fee4d2b7 [id=088956f856ff4518a56cab0330d74dc5]) Alter Table RPC with a delay of 298 ms (attempt = 5)
W0619 00:12:01.503123 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e9a221a1d9814ef99a0b4ccaa04d1017: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.503196 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c1223a8bb4fc4714b962ea5a49903675 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.503343 25071 catalog_manager.cc:2891] Scheduling retry of e9a221a1d9814ef99a0b4ccaa04d1017 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 16 ms (attempt = 1)
I0619 00:12:01.503789 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0c1f35568fe24c6c8cd2b88871b4520d on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 8140397d408e47129a182fb6255a0a39)
W0619 00:12:01.503967 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e9a221a1d9814ef99a0b4ccaa04d1017: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.504184 25071 catalog_manager.cc:2891] Scheduling retry of e9a221a1d9814ef99a0b4ccaa04d1017 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 23 ms (attempt = 1)
I0619 00:12:01.504281 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:12:01.504412 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet bae6d308dae54b0394a364d2988ec6b3: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.504647 25071 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 44 ms (attempt = 1)
I0619 00:12:01.504832 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d62715e230154e2abe126ef97feec7ce on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.505260 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:12:01.505650 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a284677c4b7f46e18e25edee6024e3eb on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:01.506067 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9d44681acacf4f04b6fcfbd4c9a18e87 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.506438 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d6c3c7f783cb43d59b46502261aeced3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet 29ee1c11105e4d1e9ac74a247cd2953d)
I0619 00:12:01.506836 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 30a9d9fda8d34caeb8242c372439321b on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:12:01.507213 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2654c7e3655d46b0bac8251218b538e3 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:12:01.507596 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0a7470bfeab041f9895e491231b57eb7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
I0619 00:12:01.508159 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4c653a255984d3bac88df63362b9334 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:12:01.508549 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet edfdedbd4eca4b62be24b1649f820d0e on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.509275 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b34043f14f1c4edd97a1d4689ced5b32 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:54 PDT)
I0619 00:12:01.509651 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54373917567a479d93dc8a3fd006d2fe on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:51 PDT)
I0619 00:12:01.510116 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 981d862d1bab4b14bb10e6fc2174e1cc on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.510498 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.510891 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8e3a356933e749b9a4ade476ec30f475 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet beddf1523a854f5b9292c0e4013d7436)
I0619 00:12:01.511261 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f24f9ab30ad04f39b5ac27e557606ab1 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:57 PDT)
I0619 00:12:01.511643 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9e5e691e6d374cd4bba1f18d8c821723 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:58 PDT)
W0619 00:12:01.504876 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 756057a7c34a4c61a261deaf849e433a: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.511965 25071 catalog_manager.cc:2891] Scheduling retry of 756057a7c34a4c61a261deaf849e433a Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 34 ms (attempt = 2)
I0619 00:12:01.512150 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d3c7114381c0423982b9ccbff120c2a2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
W0619 00:12:01.512202 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 93930ffe285d48d080a8ca4793e871d9: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.512426 25071 catalog_manager.cc:2891] Scheduling retry of 93930ffe285d48d080a8ca4793e871d9 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 26 ms (attempt = 1)
I0619 00:12:01.512662 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d0624684b6b04ba3be260b2705dc00e1 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (replaced by tablet c2cf56dada3c41d4920a4091a5fdfded)
I0619 00:12:01.513118 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6040e369e0fb445fb993d12dd883039b on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
W0619 00:12:01.512676 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e0070a115e8540cda5f999394deac616: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.513396 25071 catalog_manager.cc:2891] Scheduling retry of e0070a115e8540cda5f999394deac616 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 63 ms (attempt = 1)
I0619 00:12:01.513586 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:12:01.513649 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.513892 25071 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 47 ms (attempt = 1)
I0619 00:12:01.514104 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ef1566aa89b24620af70f2322df0dd56 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:12:00 PDT)
W0619 00:12:01.514117 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.514550 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 54adb588955b45b79de9752f254728ea on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:52 PDT)
I0619 00:12:01.514762 25071 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 60 ms (attempt = 1)
W0619 00:12:01.514989 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 2a0b025496ee4087acadc27bbb6d4983: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.515049 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:12:01.515205 25071 catalog_manager.cc:2891] Scheduling retry of 2a0b025496ee4087acadc27bbb6d4983 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 42 ms (attempt = 1)
W0619 00:12:01.515427 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet f2d15bd8d6ce460c8422625ba060c6e0: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.515530 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 17e165e0995a4c029aa31392b9bbc04a on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.515666 25071 catalog_manager.cc:2891] Scheduling retry of f2d15bd8d6ce460c8422625ba060c6e0 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 27 ms (attempt = 1)
I0619 00:12:01.516074 25067 catalog_manager.cc:2891] Scheduling retry of dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC with a delay of 43 ms (attempt = 2)
W0619 00:12:01.516297 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e5b3be4f05474541b222b0c42884ec80: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.516379 25067 catalog_manager.cc:2910] Async tablet task dba1210d9c22454f8c9e0b0293583f0c (table default.table_3ad5f49b495d4a738b79181172cfd9f4 [id=040c5029d5054cc48c9f066058cb6a45]) Alter Table RPC failed: Not found: failed to reset TS proxy: No leader found: Tablet dba1210d9c22454f8c9e0b0293583f0c consensus state has no leader
I0619 00:12:01.516517 25071 catalog_manager.cc:2891] Scheduling retry of e5b3be4f05474541b222b0c42884ec80 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.516772 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.516806 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5c3203e9b6d423f810e1106369da3d7 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
I0619 00:12:01.517005 25071 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 64 ms (attempt = 1)
W0619 00:12:01.517231 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 9d44681acacf4f04b6fcfbd4c9a18e87: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.517309 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1dd3f7afe05f49199b18220f9c11ccd2 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:53 PDT)
I0619 00:12:01.517446 25071 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 34 ms (attempt = 1)
W0619 00:12:01.517693 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 6040e369e0fb445fb993d12dd883039b: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.517812 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7f60ea5c5def47ee846d0186bbc90c18 on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:50 PDT)
I0619 00:12:01.518270 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 0ef73ff0abf44e5aa66cee31180c083c on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:55 PDT)
I0619 00:12:01.518651 25067 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3e7724f967ae41d7b523055d1a4f58db on c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951) (Table deleted at 2018-06-19 00:11:56 PDT)
W0619 00:12:01.519124 25067 connection.cc:512] server connection from 127.24.119.2:60771 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:01.519556 25067 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0619 00:12:01.519994 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 48eb7be170a64eb1bd869eba7067843d: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.520251 25067 catalog_manager.cc:2891] Scheduling retry of 48eb7be170a64eb1bd869eba7067843d Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 53 ms (attempt = 2)
W0619 00:12:01.520485 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.520748 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 42 ms (attempt = 2)
W0619 00:12:01.520982 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet f2d15bd8d6ce460c8422625ba060c6e0: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.521267 25067 catalog_manager.cc:2891] Scheduling retry of f2d15bd8d6ce460c8422625ba060c6e0 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 53 ms (attempt = 2)
W0619 00:12:01.521498 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1cfd335c18c047fba2facc5194807abb: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.521745 25067 catalog_manager.cc:2891] Scheduling retry of 1cfd335c18c047fba2facc5194807abb Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 72 ms (attempt = 2)
W0619 00:12:01.521975 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0a7470bfeab041f9895e491231b57eb7: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.522202 25067 catalog_manager.cc:2891] Scheduling retry of 0a7470bfeab041f9895e491231b57eb7 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 69 ms (attempt = 2)
W0619 00:12:01.522512 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 18dbdb33a25e487ab87ecb85af31fd55: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.522765 25067 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 61 ms (attempt = 2)
W0619 00:12:01.522996 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 756057a7c34a4c61a261deaf849e433a: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.523223 25067 catalog_manager.cc:2891] Scheduling retry of 756057a7c34a4c61a261deaf849e433a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 62 ms (attempt = 2)
W0619 00:12:01.523448 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.523697 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 69 ms (attempt = 2)
I0619 00:12:01.517909 25071 catalog_manager.cc:2891] Scheduling retry of 6040e369e0fb445fb993d12dd883039b Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 43 ms (attempt = 1)
W0619 00:12:01.523931 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 7f60ea5c5def47ee846d0186bbc90c18: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
W0619 00:12:01.524049 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.524220 25067 catalog_manager.cc:2891] Scheduling retry of 7f60ea5c5def47ee846d0186bbc90c18 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 68 ms (attempt = 2)
W0619 00:12:01.524533 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 0fee3be144c14edab6db58c99e3cda7d: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.524273 25071 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 45 ms (attempt = 1)
I0619 00:12:01.524793 25067 catalog_manager.cc:2891] Scheduling retry of 0fee3be144c14edab6db58c99e3cda7d Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 45 ms (attempt = 2)
W0619 00:12:01.524935 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.525158 25071 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 29 ms (attempt = 1)
W0619 00:12:01.525084 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 54adb588955b45b79de9752f254728ea: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.525413 25067 catalog_manager.cc:2891] Scheduling retry of 54adb588955b45b79de9752f254728ea Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 68 ms (attempt = 2)
W0619 00:12:01.525733 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 4feb648bb30247b49a4abd49da95ea7d: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
W0619 00:12:01.525406 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 1cfd335c18c047fba2facc5194807abb: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.525965 25067 catalog_manager.cc:2891] Scheduling retry of 4feb648bb30247b49a4abd49da95ea7d Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 56 ms (attempt = 2)
W0619 00:12:01.526274 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1dd3f7afe05f49199b18220f9c11ccd2: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.526505 25067 catalog_manager.cc:2891] Scheduling retry of 1dd3f7afe05f49199b18220f9c11ccd2 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 60 ms (attempt = 2)
I0619 00:12:01.526048 25071 catalog_manager.cc:2891] Scheduling retry of 1cfd335c18c047fba2facc5194807abb Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 25 ms (attempt = 1)
W0619 00:12:01.526793 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 8e3a356933e749b9a4ade476ec30f475: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.526793 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e0070a115e8540cda5f999394deac616: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.527046 25071 catalog_manager.cc:2891] Scheduling retry of 8e3a356933e749b9a4ade476ec30f475 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 56 ms (attempt = 2)
I0619 00:12:01.527107 25067 catalog_manager.cc:2891] Scheduling retry of e0070a115e8540cda5f999394deac616 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 48 ms (attempt = 2)
W0619 00:12:01.527283 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 3e7c4b61f5314e40ad71371155a50b05: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.527395 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet e0070a115e8540cda5f999394deac616: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.527716 25067 catalog_manager.cc:2891] Scheduling retry of e0070a115e8540cda5f999394deac616 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 61 ms (attempt = 2)
I0619 00:12:01.527504 25071 catalog_manager.cc:2891] Scheduling retry of 3e7c4b61f5314e40ad71371155a50b05 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 41 ms (attempt = 2)
W0619 00:12:01.527951 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fb8e94875a8347a0a473c95a77c38795: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.528223 25067 catalog_manager.cc:2891] Scheduling retry of fb8e94875a8347a0a473c95a77c38795 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 59 ms (attempt = 2)
W0619 00:12:01.528522 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet fe93c5dd62d94bdda099b9131482c033: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
W0619 00:12:01.528215 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 1c4c69294cb945938004e5b9a0550e72: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.528776 25067 catalog_manager.cc:2891] Scheduling retry of fe93c5dd62d94bdda099b9131482c033 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 40 ms (attempt = 2)
I0619 00:12:01.528872 25071 catalog_manager.cc:2891] Scheduling retry of 1c4c69294cb945938004e5b9a0550e72 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 26 ms (attempt = 1)
W0619 00:12:01.529111 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet fe93c5dd62d94bdda099b9131482c033: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.529389 25071 catalog_manager.cc:2891] Scheduling retry of fe93c5dd62d94bdda099b9131482c033 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 37 ms (attempt = 1)
W0619 00:12:01.529654 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 1cfd335c18c047fba2facc5194807abb: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.529884 25071 catalog_manager.cc:2891] Scheduling retry of 1cfd335c18c047fba2facc5194807abb Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 50 ms (attempt = 1)
W0619 00:12:01.530120 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 8ca3a0771b9e41eeb0d14d886f970159: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.530342 25071 catalog_manager.cc:2891] Scheduling retry of 8ca3a0771b9e41eeb0d14d886f970159 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 55 ms (attempt = 2)
W0619 00:12:01.530584 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 92fc5418174246bba92169339f264d62: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.530841 25071 catalog_manager.cc:2891] Scheduling retry of 92fc5418174246bba92169339f264d62 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 51 ms (attempt = 1)
W0619 00:12:01.531080 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 18dbdb33a25e487ab87ecb85af31fd55: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.531306 25071 catalog_manager.cc:2891] Scheduling retry of 18dbdb33a25e487ab87ecb85af31fd55 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 45 ms (attempt = 1)
W0619 00:12:01.531543 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e60ecc51c92c461889b360dde1dc5cf2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.531795 25071 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 40 ms (attempt = 1)
W0619 00:12:01.532032 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet e60ecc51c92c461889b360dde1dc5cf2: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:01.532258 25071 catalog_manager.cc:2891] Scheduling retry of e60ecc51c92c461889b360dde1dc5cf2 Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 19 ms (attempt = 1)
W0619 00:12:01.532490 25071 catalog_manager.cc:2855] TS 10bda0b4856f4441b5aed24cdc4764a8 (127.24.119.2:38512): Delete Tablet RPC failed for tablet 48eb7be170a64eb1bd869eba7067843d: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0619 00:12:01.533663 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet bae6d308dae54b0394a364d2988ec6b3: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.539912 25067 catalog_manager.cc:2891] Scheduling retry of bae6d308dae54b0394a364d2988ec6b3 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 40 ms (attempt = 2)
W0619 00:12:01.540125 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 075076aaa16e4ea8a596a7a8c52f103e: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.540310 25067 catalog_manager.cc:2891] Scheduling retry of 075076aaa16e4ea8a596a7a8c52f103e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 77 ms (attempt = 2)
W0619 00:12:01.540513 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 17e165e0995a4c029aa31392b9bbc04a: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.540748 25067 catalog_manager.cc:2891] Scheduling retry of 17e165e0995a4c029aa31392b9bbc04a Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 47 ms (attempt = 2)
W0619 00:12:01.540951 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9e5e691e6d374cd4bba1f18d8c821723: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.541139 25067 catalog_manager.cc:2891] Scheduling retry of 9e5e691e6d374cd4bba1f18d8c821723 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 64 ms (attempt = 2)
W0619 00:12:01.541342 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet eb36a22cf80841b58fd5ce9e6998bf00: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.541528 25067 catalog_manager.cc:2891] Scheduling retry of eb36a22cf80841b58fd5ce9e6998bf00 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 70 ms (attempt = 2)
W0619 00:12:01.541741 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet edfdedbd4eca4b62be24b1649f820d0e: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.541939 25067 catalog_manager.cc:2891] Scheduling retry of edfdedbd4eca4b62be24b1649f820d0e Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 35 ms (attempt = 2)
W0619 00:12:01.542137 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 1b40a55533f74f5eb90497d3c1064034: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.542330 25067 catalog_manager.cc:2891] Scheduling retry of 1b40a55533f74f5eb90497d3c1064034 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 43 ms (attempt = 2)
W0619 00:12:01.542526 25067 catalog_manager.cc:2855] TS c007a00e65ad41a79da1a0c1050c55b4 (127.24.119.1:32951): Delete Tablet RPC failed for tablet 9d44681acacf4f04b6fcfbd4c9a18e87: Network error: Client connection negotiation failed: client connection to 127.24.119.1:32951: connect: Connection refused (error 111)
I0619 00:12:01.542738 25067 catalog_manager.cc:2891] Scheduling retry of 9d44681acacf4f04b6fcfbd4c9a18e87 Delete Tablet RPC for TS=c007a00e65ad41a79da1a0c1050c55b4 with a delay of 50 ms (attempt = 2)
I0619 00:12:01.545714 25071 catalog_manager.cc:2891] Scheduling retry of 48eb7be170a64eb1bd869eba7067843d Delete Tablet RPC for TS=10bda0b4856f4441b5aed24cdc4764a8 with a delay of 58 ms (attempt = 1)
W0619 00:12:01.546160 25196 connection.cc:512] server connection from 127.24.119.3:55751 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.3:55751): Connection reset by peer (error 104)
I0619 00:12:01.547219 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 25059
W0619 00:12:01.563391 25892 connection.cc:512] server connection from 127.24.119.3:37305 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.3:37305): Connection reset by peer (error 104)
I0619 00:12:01.571094 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 25748
I0619 00:12:01.587385 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 25185
I0619 00:12:01.603381 25052 master-stress-test.cc:476] Tables created: 166
I0619 00:12:01.603530 25052 master-stress-test.cc:477] Tables altered: 84
I0619 00:12:01.603665 25052 master-stress-test.cc:478] Tables deleted: 82
I0619 00:12:01.603777 25052 master-stress-test.cc:479] Tablets replaced: 29
I0619 00:12:01.603883 25052 master-stress-test.cc:480] Masters restarted: 1
[       OK ] HmsConfigurations/MasterStressTest.Test/0 (18313 ms)
[ RUN      ] HmsConfigurations/MasterStressTest.Test/1
I0619 00:12:01.669018 25052 test_util.cc:208] Using random seed: 1712203653
2018-06-19 00:12:02: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Dhive.log.level=WARN -Dhive.root.logger=console -Dderby.locks.deadlockTimeout=1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/apache-hive-c0a57a33b44d73fbe0f95256f0a3822bd3720aa8-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/hadoop-2.8.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
2018-06-19T00:12:09,600  WARN [main] metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.4.0
2018-06-19T00:12:09,602  WARN [main] metastore.ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.4.0, comment = Set by MetaStore jenkins@172.28.193.241
2018-06-19T00:12:09,609  WARN [main] metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
2018-06-19T00:12:09,738  WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I0619 00:12:10.853444 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.62:58195
--webserver_interface=127.24.119.62
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_sasl_enabled=false with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:11.010524 26513 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:12:11.010879 26513 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:12:11.020092 26513 flags.cc:406] Enabled experimental flag: --hive_metastore_sasl_enabled=false
W0619 00:12:11.020203 26513 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:55188
W0619 00:12:11.020306 26513 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:12:11.020401 26513 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0619 00:12:11.020506 26513 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:12:11.020628 26513 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:12:11.020750 26513 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:12:11.040879 26513 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.62:58195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.119.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:11.049988 26513 master_main.cc:80] Initializing master server...
I0619 00:12:11.052202 26513 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 453913us
I0619 00:12:11.052927 26513 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:11.053050 26513 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:11.053295 26513 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:12:11.053405 26513 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:12:11.062137 26513 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data/instance:
uuid: "90b46df3f26b4096876d341a82aaf7c7"
format_stamp: "Formatted at 2018-06-19 07:12:11 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:11.062829 26513 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/wal/instance:
uuid: "90b46df3f26b4096876d341a82aaf7c7"
format_stamp: "Formatted at 2018-06-19 07:12:11 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:11.084828 26513 fs_manager.cc:501] Time spent creating directory manager: real 0.022s	user 0.006s	sys 0.007s
I0619 00:12:11.085983 26513 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:11.087170 26513 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:11.095235 26513 fs_manager.cc:417] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I0619 00:12:11.095351 26513 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/wal
uuid: "90b46df3f26b4096876d341a82aaf7c7"
format_stamp: "Formatted at 2018-06-19 07:12:11 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:11.095564 26513 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:12:11.232044 26513 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:11.234380 26513 master_main.cc:83] Starting Master server...
I0619 00:12:11.322152 26513 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.62:58195
I0619 00:12:11.323036 26513 webserver.cc:175] Starting webserver on 127.24.119.62:0
I0619 00:12:11.323137 26513 webserver.cc:186] Document root disabled
I0619 00:12:11.324982 26513 webserver.cc:313] Webserver started. Bound to: http://127.24.119.62:57911/
I0619 00:12:11.325716 26513 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-0/data/info.pb
I0619 00:12:11.329119 26574 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:11.330014 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 26513
I0619 00:12:11.331315 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:43000
--webserver_interface=127.24.119.61
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_sasl_enabled=false with env {}
I0619 00:12:11.343467 26574 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:12:11.361743 26574 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } has no permanent_uuid. Determining permanent_uuid...
W0619 00:12:11.363831 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:43000: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:11.383255 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } attempt: 1
W0619 00:12:11.384289 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:43000: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:11.432585 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } attempt: 2
W0619 00:12:11.433763 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:43000: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:11.489806 26575 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:12:11.490144 26575 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:12:11.499442 26575 flags.cc:406] Enabled experimental flag: --hive_metastore_sasl_enabled=false
W0619 00:12:11.499554 26575 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:55188
W0619 00:12:11.499673 26575 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:12:11.499766 26575 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0619 00:12:11.499871 26575 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:12:11.499963 26575 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:12:11.500069 26575 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:12:11.520447 26575 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:43000
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.119.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:11.529405 26575 master_main.cc:80] Initializing master server...
I0619 00:12:11.532018 26575 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 453913us
I0619 00:12:11.532734 26575 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:11.532856 26575 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:11.533095 26575 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:12:11.533202 26575 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:12:11.541988 26575 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/instance:
uuid: "b8582cfc277d4d4597c5605b7ac0ed48"
format_stamp: "Formatted at 2018-06-19 07:12:11 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:11.542682 26575 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal/instance:
uuid: "b8582cfc277d4d4597c5605b7ac0ed48"
format_stamp: "Formatted at 2018-06-19 07:12:11 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:11.545109 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } attempt: 3
W0619 00:12:11.546272 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:43000: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:11.571213 26575 fs_manager.cc:501] Time spent creating directory manager: real 0.028s	user 0.007s	sys 0.005s
I0619 00:12:11.572393 26575 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:11.573606 26575 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:11.581637 26575 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.002s
I0619 00:12:11.581763 26575 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
uuid: "b8582cfc277d4d4597c5605b7ac0ed48"
format_stamp: "Formatted at 2018-06-19 07:12:11 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:11.581977 26575 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:12:11.722591 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } attempt: 4
W0619 00:12:11.724351 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.61:43000: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:11.751121 26575 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:11.753407 26575 master_main.cc:83] Starting Master server...
I0619 00:12:11.838035 26575 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.61:43000
I0619 00:12:11.839177 26575 webserver.cc:175] Starting webserver on 127.24.119.61:0
I0619 00:12:11.839288 26575 webserver.cc:186] Document root disabled
I0619 00:12:11.841125 26575 webserver.cc:313] Webserver started. Bound to: http://127.24.119.61:40826/
I0619 00:12:11.841861 26575 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/info.pb
I0619 00:12:11.845252 26638 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:11.848563 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 26575
I0619 00:12:11.849443 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.60:33918
--webserver_interface=127.24.119.60
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_sasl_enabled=false with env {}
I0619 00:12:11.857442 26638 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:12:11.872200 26638 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:12:11.883119 26638 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } has no permanent_uuid. Determining permanent_uuid...
W0619 00:12:11.884361 26638 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:11.903694 26638 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 1
W0619 00:12:11.904867 26638 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:11.953186 26638 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 2
W0619 00:12:11.954438 26638 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:12.008541 26639 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:12:12.008909 26639 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:12:12.018314 26639 flags.cc:406] Enabled experimental flag: --hive_metastore_sasl_enabled=false
W0619 00:12:12.018445 26639 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:55188
W0619 00:12:12.018568 26639 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:12:12.018692 26639 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0619 00:12:12.018821 26639 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:12:12.018932 26639 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:12:12.019060 26639 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:12:12.027680 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } attempt: 5
I0619 00:12:12.034904 26574 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } has no permanent_uuid. Determining permanent_uuid...
W0619 00:12:12.036433 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:12.041100 26639 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.60:33918
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.119.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:12.050148 26639 master_main.cc:80] Initializing master server...
I0619 00:12:12.051997 26639 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 454413us
I0619 00:12:12.052762 26639 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:12.052893 26639 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:12.053143 26639 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:12:12.053261 26639 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:12:12.062113 26639 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/instance:
uuid: "555bff44d4f942eaad13f790cca0a696"
format_stamp: "Formatted at 2018-06-19 07:12:12 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:12.062853 26639 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal/instance:
uuid: "555bff44d4f942eaad13f790cca0a696"
format_stamp: "Formatted at 2018-06-19 07:12:12 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:12.065765 26638 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 3
W0619 00:12:12.066970 26638 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:12.074828 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 1
W0619 00:12:12.076161 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:12.078006 26639 fs_manager.cc:501] Time spent creating directory manager: real 0.015s	user 0.009s	sys 0.005s
I0619 00:12:12.079186 26639 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:12.080353 26639 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:12.088364 26639 fs_manager.cc:417] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I0619 00:12:12.088485 26639 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
uuid: "555bff44d4f942eaad13f790cca0a696"
format_stamp: "Formatted at 2018-06-19 07:12:12 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:12.088728 26639 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:12:12.124512 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 2
W0619 00:12:12.125810 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:12.226166 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 3
W0619 00:12:12.227526 26574 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:12.239908 26639 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:12.242290 26639 master_main.cc:83] Starting Master server...
I0619 00:12:12.243265 26638 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 4
W0619 00:12:12.244714 26638 consensus_peers.cc:566] Error getting permanent uuid from config peer 127.24.119.60:33918: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:12.329648 26639 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.60:33918
I0619 00:12:12.330468 26639 webserver.cc:175] Starting webserver on 127.24.119.60:0
I0619 00:12:12.330591 26639 webserver.cc:186] Document root disabled
I0619 00:12:12.332367 26639 webserver.cc:313] Webserver started. Bound to: http://127.24.119.60:32921/
I0619 00:12:12.333112 26639 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/info.pb
I0619 00:12:12.336549 26702 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:12.337447 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 26639
I0619 00:12:12.338745 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.24.119.1:0
--local_ip_for_outbound_sockets=127.24.119.1
--webserver_interface=127.24.119.1
--webserver_port=0
--tserver_master_addrs=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0619 00:12:12.348362 26702 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:12:12.363016 26702 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:12:12.367911 26574 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 4
I0619 00:12:12.369837 26702 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } has no permanent_uuid. Determining permanent_uuid...
I0619 00:12:12.389703 26702 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Bootstrap starting.
I0619 00:12:12.390286 26574 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Bootstrap starting.
I0619 00:12:12.392014 26702 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: No blocks or log segments found. Creating new log.
I0619 00:12:12.393267 26574 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: No blocks or log segments found. Creating new log.
I0619 00:12:12.393369 26702 log.cc:525] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Log is configured to *not* fsync() on all Append() calls
I0619 00:12:12.395138 26574 log.cc:525] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Log is configured to *not* fsync() on all Append() calls
I0619 00:12:12.396971 26702 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: No bootstrap required, opened a new log
I0619 00:12:12.398974 26574 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: No bootstrap required, opened a new log
I0619 00:12:12.407490 26702 raft_consensus.cc:325] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.408007 26702 raft_consensus.cc:351] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:12.408185 26702 raft_consensus.cc:628] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555bff44d4f942eaad13f790cca0a696, State: Initialized, Role: FOLLOWER
I0619 00:12:12.408747 26702 consensus_queue.cc:227] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.411638 26702 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.412302 26702 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.413933 26702 sys_catalog.cc:424] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:12:12.416726 26574 raft_consensus.cc:325] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.417441 26574 raft_consensus.cc:351] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:12.417665 26574 raft_consensus.cc:628] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b46df3f26b4096876d341a82aaf7c7, State: Initialized, Role: FOLLOWER
I0619 00:12:12.418388 26574 consensus_queue.cc:227] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.421291 26574 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.421995 26574 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.422511 26711 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.423252 26711 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.411913 26709 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.506305 26709 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.423425 26574 sys_catalog.cc:424] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: configured and running, proceeding with master startup.
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:12.526801 26703 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:12:12.527395 26703 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:12:12.542896 26703 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:12:12.543448 26703 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.119.1
W0619 00:12:12.487762 26530 tablet.cc:1678] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:12:12.548061 26638 consensus_peers.cc:576] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } attempt: 5
I0619 00:12:12.562706 26638 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Bootstrap starting.
I0619 00:12:12.563052 26713 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:12.565816 26638 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: No blocks or log segments found. Creating new log.
W0619 00:12:12.495393 26658 tablet.cc:1678] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:12:12.567564 26638 log.cc:525] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Log is configured to *not* fsync() on all Append() calls
I0619 00:12:12.572268 26638 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: No bootstrap required, opened a new log
I0619 00:12:12.572254 26716 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:12.573330 26513 master_main.cc:86] Master server successfully started.
I0619 00:12:12.574298 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:12.578950 26703 tablet_server_main.cc:78] Tablet server non-default flags:
--log_preallocate_segments=false
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.24.119.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.119.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.119.1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:12.579284 26721 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:12.580304 26639 master_main.cc:86] Master server successfully started.
I0619 00:12:12.588515 26638 raft_consensus.cc:325] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.589504 26638 raft_consensus.cc:351] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:12.589680 26638 raft_consensus.cc:628] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8582cfc277d4d4597c5605b7ac0ed48, State: Initialized, Role: FOLLOWER
I0619 00:12:12.590209 26703 tablet_server_main.cc:85] Initializing tablet server...
I0619 00:12:12.590224 26638 consensus_queue.cc:227] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.592557 26703 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 454413us
I0619 00:12:12.593049 26703 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:12.593183 26703 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:12.592726 26638 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.593469 26638 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.593508 26703 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:12:12.593657 26703 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:12:12.594624 26638 sys_catalog.cc:424] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:12:12.607113 26703 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data/instance:
uuid: "10c4e713d4c44dd8904e5f77ba815785"
format_stamp: "Formatted at 2018-06-19 07:12:12 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:12.608119 26703 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/instance:
uuid: "10c4e713d4c44dd8904e5f77ba815785"
format_stamp: "Formatted at 2018-06-19 07:12:12 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:12.638427 26703 fs_manager.cc:501] Time spent creating directory manager: real 0.030s	user 0.013s	sys 0.007s
I0619 00:12:12.640209 26703 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:12.641676 26703 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:12.642707 26711 raft_consensus.cc:457] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:12.643007 26711 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Snoozing failure detection for 0.532s (starting election)
I0619 00:12:12.643290 26711 raft_consensus.cc:479] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.644558 26711 leader_election.cc:243] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:12.645149 26711 leader_election.cc:243] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:12.646637 26679 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90b46df3f26b4096876d341a82aaf7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696" is_pre_election: true
I0619 00:12:12.647267 26679 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.613s (vote granted)
I0619 00:12:12.647532 26679 raft_consensus.cc:2245] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b46df3f26b4096876d341a82aaf7c7 in term 0.
I0619 00:12:12.648530 26615 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90b46df3f26b4096876d341a82aaf7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
I0619 00:12:12.649176 26615 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.620s (vote granted)
I0619 00:12:12.649456 26615 raft_consensus.cc:2245] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b46df3f26b4096876d341a82aaf7c7 in term 0.
I0619 00:12:12.650354 26524 leader_election.cc:379] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:12.650657 26524 leader_election.cc:270] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:12.650995 26523 leader_election.cc:379] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:12.651283 26711 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Snoozing failure detection for 0.668s (election complete)
I0619 00:12:12.651525 26711 raft_consensus.cc:2539] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:12.651757 26711 raft_consensus.cc:457] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:12.651942 26711 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Snoozing failure detection for 0.646s (starting election)
I0619 00:12:12.652137 26711 raft_consensus.cc:2784] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:12.656890 26711 raft_consensus.cc:479] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.657866 26711 leader_election.cc:243] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 election: Requesting vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:12.658571 26711 leader_election.cc:243] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 election: Requesting vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:12.658900 26615 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90b46df3f26b4096876d341a82aaf7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48"
I0619 00:12:12.659320 26615 raft_consensus.cc:2784] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:12.659592 26615 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.609s (vote granted)
I0619 00:12:12.661010 26679 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90b46df3f26b4096876d341a82aaf7c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696"
I0619 00:12:12.661336 26679 raft_consensus.cc:2784] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:12.661527 26679 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.538s (vote granted)
I0619 00:12:12.593021 26722 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.663208 26722 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.665999 26679 raft_consensus.cc:2245] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b46df3f26b4096876d341a82aaf7c7 in term 1.
I0619 00:12:12.666904 26523 leader_election.cc:379] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 election: Vote granted by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:12.667148 26523 leader_election.cc:270] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:12.667600 26711 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Snoozing failure detection for 0.614s (election complete)
I0619 00:12:12.667819 26711 raft_consensus.cc:2539] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:12.668921 26703 fs_manager.cc:417] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I0619 00:12:12.669088 26703 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal
uuid: "10c4e713d4c44dd8904e5f77ba815785"
format_stamp: "Formatted at 2018-06-19 07:12:12 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:12.669478 26703 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:12:12.671676 26615 raft_consensus.cc:2245] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b46df3f26b4096876d341a82aaf7c7 in term 1.
I0619 00:12:12.672425 26524 leader_election.cc:379] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [CANDIDATE]: Term 1 election: Vote granted by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:12.672557 26727 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:12.674098 26729 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:12.674525 26711 raft_consensus.cc:593] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Becoming Leader. State: Replica: 90b46df3f26b4096876d341a82aaf7c7, State: Running, Role: LEADER
I0619 00:12:12.675545 26575 master_main.cc:86] Master server successfully started.
I0619 00:12:12.675376 26711 consensus_queue.cc:204] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:12.683702 26728 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b46df3f26b4096876d341a82aaf7c7. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.684192 26728 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:12.687561 26731 catalog_manager.cc:1006] Loading table and tablet metadata into memory...
I0619 00:12:12.693845 26731 catalog_manager.cc:1017] Initializing Kudu internal certificate authority...
I0619 00:12:12.732239 26679 raft_consensus.cc:1081] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Refusing update from remote peer 90b46df3f26b4096876d341a82aaf7c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:12:12.732568 26615 raft_consensus.cc:1081] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Refusing update from remote peer 90b46df3f26b4096876d341a82aaf7c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0619 00:12:12.733808 26711 consensus_queue.cc:963] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:12.734331 26728 consensus_queue.cc:963] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:12.765166 26594 tablet.cc:1678] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:12:12.769541 26709 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b46df3f26b4096876d341a82aaf7c7. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.770221 26709 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.773792 26722 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b46df3f26b4096876d341a82aaf7c7. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.774483 26722 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.792668 26709 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.793275 26709 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.794569 26722 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.795377 26722 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:12.796283 26731 catalog_manager.cc:904] Generated new certificate authority record
I0619 00:12:12.797089 26728 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.797767 26728 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:12.798408 26731 catalog_manager.cc:1026] Loading token signing keys...
I0619 00:12:12.806507 26711 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:12.807118 26711 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:12.824380 26731 catalog_manager.cc:4090] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Generated new TSK 0
I0619 00:12:12.831616 26731 catalog_manager.cc:1037] Loading latest processed Hive Metastore notification log event ID...
I0619 00:12:12.887883 26703 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:12.891414 26703 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0619 00:12:12.893708 26703 tablet_server_main.cc:90] Starting tablet server...
I0619 00:12:13.118715 26703 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.1:42295
I0619 00:12:13.119654 26703 webserver.cc:175] Starting webserver on 127.24.119.1:0
I0619 00:12:13.119779 26703 webserver.cc:186] Document root disabled
I0619 00:12:13.121969 26703 webserver.cc:313] Webserver started. Bound to: http://127.24.119.1:50497/
I0619 00:12:13.122634 26703 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/data/info.pb
I0619 00:12:13.131956 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 26703
I0619 00:12:13.133374 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data
--block_manager=log
--rpc_bind_addresses=127.24.119.2:0
--local_ip_for_outbound_sockets=127.24.119.2
--webserver_interface=127.24.119.2
--webserver_port=0
--tserver_master_addrs=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0619 00:12:13.158327 26703 tablet_server_main.cc:93] Tablet server successfully started.
I0619 00:12:13.214443 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:13.214802 26856 heartbeater.cc:426] Registering TS with master...
I0619 00:12:13.215682 26856 heartbeater.cc:475] Master 127.24.119.61:43000 requested a full tablet report, sending...
I0619 00:12:13.218088 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:13.218312 26857 heartbeater.cc:426] Registering TS with master...
I0619 00:12:13.218732 26857 heartbeater.cc:475] Master 127.24.119.60:33918 requested a full tablet report, sending...
I0619 00:12:13.218823 26605 ts_manager.cc:81] Registered new tserver with Master: 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:13.221207 26669 ts_manager.cc:81] Registered new tserver with Master: 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:13.215119 26855 heartbeater.cc:346] Connected to a master server at 127.24.119.62:58195
I0619 00:12:13.225772 26855 heartbeater.cc:426] Registering TS with master...
I0619 00:12:13.226584 26855 heartbeater.cc:475] Master 127.24.119.62:58195 requested a full tablet report, sending...
I0619 00:12:13.228981 26541 ts_manager.cc:81] Registered new tserver with Master: 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:13.231767 26541 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.24.119.1:49211
I0619 00:12:13.336796 26866 raft_consensus.cc:457] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:13.337128 26866 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.666s (starting election)
I0619 00:12:13.337397 26866 raft_consensus.cc:479] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:13.338769 26866 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:13.339308 26866 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:13.340281 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:13.340510 26615 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
I0619 00:12:13.340945 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.341091 26615 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.342389 26653 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000). Message: Invalid argument: T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.342792 26650 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.343065 26650 leader_election.cc:270] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:13.343572 26866 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.566s (election complete)
I0619 00:12:13.343857 26866 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:13.349761 26859 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:12:13.350155 26859 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:12:13.358316 26859 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:12:13.358671 26859 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.119.2
I0619 00:12:13.376986 26859 tablet_server_main.cc:78] Tablet server non-default flags:
--log_preallocate_segments=false
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.24.119.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.119.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.119.2
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:13.386099 26859 tablet_server_main.cc:85] Initializing tablet server...
I0619 00:12:13.388664 26859 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 454913us
I0619 00:12:13.389124 26859 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:13.389255 26859 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:13.389510 26859 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:12:13.389629 26859 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:12:13.398536 26859 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data/instance:
uuid: "1f719e8be9c9447a9acd40728c0a5710"
format_stamp: "Formatted at 2018-06-19 07:12:13 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:13.399317 26859 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/instance:
uuid: "1f719e8be9c9447a9acd40728c0a5710"
format_stamp: "Formatted at 2018-06-19 07:12:13 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:13.414274 26859 fs_manager.cc:501] Time spent creating directory manager: real 0.015s	user 0.010s	sys 0.003s
I0619 00:12:13.415451 26859 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:13.416653 26859 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:13.425091 26859 fs_manager.cc:417] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I0619 00:12:13.425218 26859 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal
uuid: "1f719e8be9c9447a9acd40728c0a5710"
format_stamp: "Formatted at 2018-06-19 07:12:13 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:13.425437 26859 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:12:13.575410 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:13.602676 26721 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: acquiring CA information for follower catalog manager: success
I0619 00:12:13.605459 26721 catalog_manager.cc:1090] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0619 00:12:13.646489 26859 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:13.666041 26859 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0619 00:12:13.673074 26859 tablet_server_main.cc:90] Starting tablet server...
I0619 00:12:13.673732 26727 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:13.688899 26729 catalog_manager.cc:1062] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: acquiring CA information for follower catalog manager: success
I0619 00:12:13.696110 26729 catalog_manager.cc:1090] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0619 00:12:13.735494 26855 heartbeater.cc:467] Master 127.24.119.62:58195 was elected leader, sending a full tablet report...
I0619 00:12:13.787729 26896 raft_consensus.cc:457] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:13.792713 26896 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.568s (starting election)
I0619 00:12:13.793100 26896 raft_consensus.cc:479] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:13.796049 26896 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:13.798264 26896 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:13.798396 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:13.798996 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.799932 26678 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696" is_pre_election: true
I0619 00:12:13.800524 26678 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.800776 26586 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.801293 26587 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918). Message: Invalid argument: T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.801553 26587 leader_election.cc:270] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:13.803663 26896 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.705s (election complete)
I0619 00:12:13.803900 26896 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:13.956195 26911 raft_consensus.cc:457] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:13.964684 26911 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.759s (starting election)
I0619 00:12:13.965052 26911 raft_consensus.cc:479] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:13.966485 26911 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:13.966956 26911 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:13.967803 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:13.968132 26615 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
I0619 00:12:13.968374 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.968703 26615 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.969213 26650 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.969487 26653 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000). Message: Invalid argument: T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:13.969759 26653 leader_election.cc:270] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:13.970196 26911 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.665s (election complete)
I0619 00:12:13.970396 26911 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:14.372819 26958 raft_consensus.cc:457] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:14.373208 26958 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.649s (starting election)
I0619 00:12:14.373534 26958 raft_consensus.cc:479] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:14.374636 26958 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:14.375107 26958 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:14.375936 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:14.376531 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.377526 26678 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696" is_pre_election: true
I0619 00:12:14.377734 26586 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.378121 26678 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.378772 26587 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918). Message: Invalid argument: T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.379032 26587 leader_election.cc:270] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:14.379465 26958 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.520s (election complete)
I0619 00:12:14.379695 26958 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:14.521572 26979 raft_consensus.cc:457] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:14.521942 26979 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 1.020s (starting election)
I0619 00:12:14.522274 26979 raft_consensus.cc:479] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:14.523387 26979 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:14.531903 26979 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:14.532758 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:14.533350 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.533124 26615 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
I0619 00:12:14.533694 26615 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.534227 26650 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.534508 26653 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000). Message: Invalid argument: T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:14.534793 26653 leader_election.cc:270] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:14.535251 26979 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.672s (election complete)
I0619 00:12:14.535475 26979 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:14.576365 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:14.606895 26859 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.2:55248
I0619 00:12:14.608466 26859 webserver.cc:175] Starting webserver on 127.24.119.2:0
I0619 00:12:14.608664 26859 webserver.cc:186] Document root disabled
I0619 00:12:14.611196 26859 webserver.cc:313] Webserver started. Bound to: http://127.24.119.2:37964/
I0619 00:12:14.612149 26859 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/data/info.pb
I0619 00:12:14.617902 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 26859
I0619 00:12:14.622707 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.24.119.3:0
--local_ip_for_outbound_sockets=127.24.119.3
--webserver_interface=127.24.119.3
--webserver_port=0
--tserver_master_addrs=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0619 00:12:14.674809 26727 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:14.699038 26859 tablet_server_main.cc:93] Tablet server successfully started.
I0619 00:12:14.721721 26995 heartbeater.cc:346] Connected to a master server at 127.24.119.62:58195
I0619 00:12:14.722043 26995 heartbeater.cc:426] Registering TS with master...
I0619 00:12:14.723184 26995 heartbeater.cc:475] Master 127.24.119.62:58195 requested a full tablet report, sending...
I0619 00:12:14.725690 26541 ts_manager.cc:81] Registered new tserver with Master: 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:14.727450 26541 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.24.119.2:56697
I0619 00:12:14.728072 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:14.728294 27000 heartbeater.cc:426] Registering TS with master...
I0619 00:12:14.728724 27000 heartbeater.cc:475] Master 127.24.119.60:33918 requested a full tablet report, sending...
I0619 00:12:14.731385 26669 ts_manager.cc:81] Registered new tserver with Master: 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:14.734689 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:14.734922 26996 heartbeater.cc:426] Registering TS with master...
I0619 00:12:14.735265 26996 heartbeater.cc:475] Master 127.24.119.61:43000 requested a full tablet report, sending...
I0619 00:12:14.737309 26605 ts_manager.cc:81] Registered new tserver with Master: 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:14.876632 26999 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:12:14.877243 26999 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0619 00:12:14.894091 26999 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:12:14.894698 26999 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.119.3
I0619 00:12:14.930071 26999 tablet_server_main.cc:78] Tablet server non-default flags:
--log_preallocate_segments=false
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.24.119.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.119.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.119.3
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:14.946348 26999 tablet_server_main.cc:85] Initializing tablet server...
I0619 00:12:14.949957 26999 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 455413us
I0619 00:12:14.950600 26999 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:14.950798 26999 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:14.951160 26999 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0619 00:12:14.951331 26999 server_base.cc:433] Attempting to create new FS layout instead
I0619 00:12:14.966516 26999 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data/instance:
uuid: "a2c1dfdaa6d442539e4342d184d2167c"
format_stamp: "Formatted at 2018-06-19 07:12:14 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:14.967736 26999 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/instance:
uuid: "a2c1dfdaa6d442539e4342d184d2167c"
format_stamp: "Formatted at 2018-06-19 07:12:14 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:14.992092 26999 fs_manager.cc:501] Time spent creating directory manager: real 0.024s	user 0.013s	sys 0.008s
I0619 00:12:14.994102 26999 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:14.995517 26999 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:15.010641 26999 fs_manager.cc:417] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I0619 00:12:15.010860 26999 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal
uuid: "a2c1dfdaa6d442539e4342d184d2167c"
format_stamp: "Formatted at 2018-06-19 07:12:14 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:15.011222 26999 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:12:15.226037 26999 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:15.232874 26995 heartbeater.cc:467] Master 127.24.119.62:58195 was elected leader, sending a full tablet report...
I0619 00:12:15.260069 26999 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0619 00:12:15.262565 26999 tablet_server_main.cc:90] Starting tablet server...
I0619 00:12:15.549706 27095 raft_consensus.cc:457] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:15.550040 27095 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.577s (starting election)
I0619 00:12:15.550345 27095 raft_consensus.cc:479] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:15.551434 27095 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:15.551908 27095 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:15.552757 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:15.553328 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:15.554162 26650 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:15.556429 26615 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
I0619 00:12:15.557009 26615 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:15.557790 26653 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000). Message: Invalid argument: T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:15.558048 26653 leader_election.cc:270] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:15.558466 27095 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.981s (election complete)
I0619 00:12:15.558722 27095 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:15.577193 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:15.662636 26999 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.3:34806
I0619 00:12:15.663554 26999 webserver.cc:175] Starting webserver on 127.24.119.3:0
I0619 00:12:15.663691 26999 webserver.cc:186] Document root disabled
I0619 00:12:15.666126 26999 webserver.cc:313] Webserver started. Bound to: http://127.24.119.3:50645/
I0619 00:12:15.666816 26999 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/data/info.pb
I0619 00:12:15.667544 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 26999
I0619 00:12:15.676054 26727 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:15.723057 26999 tablet_server_main.cc:93] Tablet server successfully started.
I0619 00:12:15.733197 27127 heartbeater.cc:346] Connected to a master server at 127.24.119.62:58195
I0619 00:12:15.733448 27127 heartbeater.cc:426] Registering TS with master...
I0619 00:12:15.734323 27127 heartbeater.cc:475] Master 127.24.119.62:58195 requested a full tablet report, sending...
I0619 00:12:15.736433 26541 ts_manager.cc:81] Registered new tserver with Master: a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:15.737835 26541 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.24.119.3:59329
I0619 00:12:15.738615 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:15.738814 27133 heartbeater.cc:426] Registering TS with master...
I0619 00:12:15.739259 27133 heartbeater.cc:475] Master 127.24.119.60:33918 requested a full tablet report, sending...
I0619 00:12:15.741916 26669 ts_manager.cc:81] Registered new tserver with Master: a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:15.734464 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:15.750946 27130 heartbeater.cc:426] Registering TS with master...
I0619 00:12:15.751231 27130 heartbeater.cc:475] Master 127.24.119.61:43000 requested a full tablet report, sending...
I0619 00:12:15.753770 26605 ts_manager.cc:81] Registered new tserver with Master: a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:15.755578 25052 external_mini_cluster.cc:490] 3 TS(s) registered with all masters
I0619 00:12:15.805694 26541 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_31d388c40f064a2a97bbbec12c77fbd9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:15.818475 26539 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_4a6e4e28e18547719a48db319fc95750"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:15.821229 26539 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4a6e4e28e18547719a48db319fc95750 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:12:15.821229 26541 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_31d388c40f064a2a97bbbec12c77fbd9 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:15.808315 26540 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_f9c29cfbe84f43ff836467ab27c1a7ca"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:15.823282 26540 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f9c29cfbe84f43ff836467ab27c1a7ca in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:15.825153 26538 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_d9d85b384d834dd9ac9814f9d0fa9ce3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:15.826113 26538 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:15.880116 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 26575
W0619 00:12:15.937332 26746 connection.cc:512] client connection to 127.24.119.61:43000 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.61:43000): Connection reset by peer (error 104)
W0619 00:12:15.937908 26880 connection.cc:512] client connection to 127.24.119.61:43000 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.61:43000): Connection reset by peer (error 104)
W0619 00:12:15.939013 27017 connection.cc:512] client connection to 127.24.119.61:43000 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.61:43000): Connection reset by peer (error 104)
I0619 00:12:16.241446 27127 heartbeater.cc:467] Master 127.24.119.62:58195 was elected leader, sending a full tablet report...
W0619 00:12:16.261926 26524 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000): Couldn't send request to peer b8582cfc277d4d4597c5605b7ac0ed48 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:16.262894 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:16.263749 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:16.263926 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.330909 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:43000
--webserver_interface=127.24.119.61
--webserver_port=40826
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_sasl_enabled=false with env {}
I0619 00:12:16.524066 27063 tablet_service.cc:759] Processing CreateTablet for tablet 98df4155785241398c7270720d8a31cd (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.524982 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 98df4155785241398c7270720d8a31cd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.532598 26930 tablet_service.cc:759] Processing CreateTablet for tablet 98df4155785241398c7270720d8a31cd (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.533520 26930 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 98df4155785241398c7270720d8a31cd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.535379 26792 tablet_service.cc:759] Processing CreateTablet for tablet 98df4155785241398c7270720d8a31cd (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.536237 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 98df4155785241398c7270720d8a31cd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.563292 27063 ts_tablet_manager.cc:1173] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.570552 26792 ts_tablet_manager.cc:1173] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.574761 26536 master_service.cc:523] ReplaceTablet: received request to replace tablet 98df4155785241398c7270720d8a31cd from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:16.577482 26537 master_service.cc:523] ReplaceTablet: received request to replace tablet 98df4155785241398c7270720d8a31cd from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:16.579530 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:16.581218 26930 ts_tablet_manager.cc:1173] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.581395 27195 ts_tablet_manager.cc:1028] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:16.585858 26536 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 98df4155785241398c7270720d8a31cd
I0619 00:12:16.586338 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.586817 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.587229 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.587591 26536 catalog_manager.cc:4596] ReplaceTablet: tablet 98df4155785241398c7270720d8a31cd deleted and replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe
I0619 00:12:16.589087 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:16.589380 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:16.591558 27196 ts_tablet_manager.cc:1028] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:16.592213 27196 tablet_bootstrap.cc:427] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:16.598671 26521 catalog_manager.cc:3083] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.598973 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=a2c1dfdaa6d442539e4342d184d2167c with a delay of 19 ms (attempt = 1)
I0619 00:12:16.599987 27196 tablet_bootstrap.cc:578] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:16.602102 27196 log.cc:525] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Log is configured to *not* fsync() on all Append() calls
I0619 00:12:16.607121 27195 tablet_bootstrap.cc:427] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:16.612818 27195 tablet_bootstrap.cc:578] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:16.618700 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.619535 27195 log.cc:525] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Log is configured to *not* fsync() on all Append() calls
I0619 00:12:16.620851 27063 tablet_service.cc:759] Processing CreateTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.621140 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:16.621461 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b1ca17b4629f4570bd99ddfbe12121fe. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:16.622815 26521 catalog_manager.cc:3083] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.623051 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=a2c1dfdaa6d442539e4342d184d2167c with a delay of 79 ms (attempt = 2)
I0619 00:12:16.624790 26792 tablet_service.cc:759] Processing CreateTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.625406 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b1ca17b4629f4570bd99ddfbe12121fe. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.630034 27063 ts_tablet_manager.cc:1173] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.634091 26792 ts_tablet_manager.cc:1173] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:16.636438 26523 catalog_manager.cc:3083] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.636709 26523 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=10c4e713d4c44dd8904e5f77ba815785 with a delay of 48 ms (attempt = 1)
I0619 00:12:16.638854 27196 tablet_bootstrap.cc:427] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:16.639279 27196 ts_tablet_manager.cc:1045] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.047s	user 0.003s	sys 0.010s
I0619 00:12:16.645560 27195 tablet_bootstrap.cc:427] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:16.645970 27195 ts_tablet_manager.cc:1045] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.064s	user 0.005s	sys 0.004s
I0619 00:12:16.657908 27196 raft_consensus.cc:325] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.658932 27196 raft_consensus.cc:351] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.659757 27196 raft_consensus.cc:628] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:16.660559 27196 consensus_queue.cc:227] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.677265 26537 catalog_manager.cc:3663] Got report from deleted tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe): Sending delete request for this tablet
I0619 00:12:16.678470 26537 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
W0619 00:12:16.679348 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.680846 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:16.680155 27195 raft_consensus.cc:325] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.681216 27195 raft_consensus.cc:351] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.681422 27195 raft_consensus.cc:628] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:16.681630 27196 ts_tablet_manager.cc:1073] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.042s	user 0.018s	sys 0.007s
I0619 00:12:16.682591 27196 ts_tablet_manager.cc:1028] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:16.682998 27196 tablet_bootstrap.cc:427] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:16.684649 26523 catalog_manager.cc:3083] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.684576 27063 tablet_service.cc:759] Processing CreateTablet for tablet 7485c30d9a814bf7b30223b93354bc22 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.684895 26523 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=10c4e713d4c44dd8904e5f77ba815785 with a delay of 19 ms (attempt = 1)
I0619 00:12:16.685115 26927 tablet_service.cc:759] Processing CreateTablet for tablet 7485c30d9a814bf7b30223b93354bc22 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.685227 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7485c30d9a814bf7b30223b93354bc22. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.685467 26523 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.685756 26927 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7485c30d9a814bf7b30223b93354bc22. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.686544 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:16.686770 27196 tablet_bootstrap.cc:578] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:16.688910 27200 tablet_replica.cc:261] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:16.689210 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.689466 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.689357 26791 tablet_service.cc:759] Processing CreateTablet for tablet 7485c30d9a814bf7b30223b93354bc22 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.689693 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.689882 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.690064 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7485c30d9a814bf7b30223b93354bc22. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.690100 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.690327 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.690507 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.690769 27200 raft_consensus.cc:2042] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.694700 27063 ts_tablet_manager.cc:1173] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.694769 26927 ts_tablet_manager.cc:1173] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.695783 27200 raft_consensus.cc:2069] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:16.682204 27195 consensus_queue.cc:227] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.702384 26791 ts_tablet_manager.cc:1173] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.703943 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.704897 27198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:16.705621 27198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:16.705370 27200 ts_tablet_manager.cc:1369] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:16.705720 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.706068 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:16.707201 26523 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
W0619 00:12:16.707494 26524 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000): Couldn't send request to peer b8582cfc277d4d4597c5605b7ac0ed48 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times.
W0619 00:12:16.707974 26521 catalog_manager.cc:3083] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.708232 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=a2c1dfdaa6d442539e4342d184d2167c with a delay of 99 ms (attempt = 3)
I0619 00:12:16.708514 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:16.709733 26536 catalog_manager.cc:3663] Got report from deleted tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe): Sending delete request for this tablet
I0619 00:12:16.710470 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.713145 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.1:39548
W0619 00:12:16.721148 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.721168 26536 catalog_manager.cc:3663] Got report from deleted tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe): Sending delete request for this tablet
I0619 00:12:16.721877 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.721922 27206 raft_consensus.cc:457] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:16.723065 27206 raft_consensus.cc:2732] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.639s (starting election)
I0619 00:12:16.723493 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:16.723426 27206 raft_consensus.cc:479] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:16.724556 26521 catalog_manager.cc:3083] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.725188 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=a2c1dfdaa6d442539e4342d184d2167c with a delay of 17 ms (attempt = 1)
I0619 00:12:16.725244 27206 leader_election.cc:243] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:16.729138 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.729807 27206 leader_election.cc:243] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:16.731374 26536 catalog_manager.cc:3663] Got report from deleted tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe): Sending delete request for this tablet
I0619 00:12:16.732115 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.732800 27195 ts_tablet_manager.cc:1073] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.087s	user 0.021s	sys 0.005s
I0619 00:12:16.733131 27195 ts_tablet_manager.cc:1028] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:16.588248 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.729260 27200 ts_tablet_manager.cc:1382] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:16.734223 27200 log.cc:986] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/98df4155785241398c7270720d8a31cd
I0619 00:12:16.735136 27197 ts_tablet_manager.cc:1028] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:16.735890 27197 tablet_bootstrap.cc:427] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:16.740408 27196 tablet_bootstrap.cc:427] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:16.740808 27196 ts_tablet_manager.cc:1045] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.058s	user 0.005s	sys 0.002s
I0619 00:12:16.740893 27197 tablet_bootstrap.cc:578] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:16.743412 27197 log.cc:525] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Log is configured to *not* fsync() on all Append() calls
I0619 00:12:16.743250 27196 raft_consensus.cc:325] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.743739 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.743971 27196 raft_consensus.cc:351] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.744144 27196 raft_consensus.cc:628] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:16.744686 27196 consensus_queue.cc:227] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.745551 27195 tablet_bootstrap.cc:427] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:16.747262 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.747323 27196 ts_tablet_manager.cc:1073] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
I0619 00:12:16.747694 27196 ts_tablet_manager.cc:1028] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:16.748384 27196 tablet_bootstrap.cc:427] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:16.748996 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.749492 27200 ts_tablet_manager.cc:1403] T 98df4155785241398c7270720d8a31cd P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:16.753489 27196 tablet_bootstrap.cc:578] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:16.754338 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:16.755578 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet 98df4155785241398c7270720d8a31cd because the tablet was not found. No further retry: Not found: Tablet not found: 98df4155785241398c7270720d8a31cd
I0619 00:12:16.757998 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:16.758328 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet 98df4155785241398c7270720d8a31cd because the tablet was not found. No further retry: Not found: Tablet not found: 98df4155785241398c7270720d8a31cd
I0619 00:12:16.758487 27199 tablet_replica.cc:261] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:16.758776 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.758990 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.759187 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.759367 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.759589 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.759795 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.759958 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.760174 27199 raft_consensus.cc:2042] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.760527 27199 raft_consensus.cc:2069] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:16.761584 27195 tablet_bootstrap.cc:578] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:16.763133 27199 ts_tablet_manager.cc:1369] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:16.764116 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.3:56160
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:16.716523 27176 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I0619 00:12:16.620795 26928 tablet_service.cc:759] Processing CreateTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:16.766140 27176 flags.cc:406] Enabled unsafe flag: --never_fsync=true
I0619 00:12:16.768224 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b1ca17b4629f4570bd99ddfbe12121fe. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:16.771643 27015 leader_election.cc:306] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Not found: Tablet not found: 98df4155785241398c7270720d8a31cd
I0619 00:12:16.774322 27199 ts_tablet_manager.cc:1382] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:16.775017 27203 raft_consensus.cc:457] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:16.775308 27203 raft_consensus.cc:2732] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.018s (starting election)
I0619 00:12:16.775616 27203 raft_consensus.cc:479] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.777292 27203 leader_election.cc:243] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:16.777848 27203 leader_election.cc:243] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
W0619 00:12:16.779892 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.780136 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 23 ms (attempt = 1)
I0619 00:12:16.768906 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "98df4155785241398c7270720d8a31cd" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:16.787648 27197 tablet_bootstrap.cc:427] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:16.788091 27197 ts_tablet_manager.cc:1045] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.053s	user 0.003s	sys 0.011s
I0619 00:12:16.788956 26928 ts_tablet_manager.cc:1173] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.790107 27196 tablet_bootstrap.cc:427] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:16.791086 27196 ts_tablet_manager.cc:1045] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.043s	user 0.007s	sys 0.004s
I0619 00:12:16.793153 27196 raft_consensus.cc:325] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.793766 27196 raft_consensus.cc:351] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.793956 27196 raft_consensus.cc:628] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:16.794391 27196 consensus_queue.cc:227] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.797490 27196 ts_tablet_manager.cc:1073] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
W0619 00:12:16.785900 27013 leader_election.cc:306] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:16.801144 27013 leader_election.cc:270] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:12:16.801612 27013 raft_consensus.cc:2442] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:12:16.802238 27199 log.cc:986] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/98df4155785241398c7270720d8a31cd
I0619 00:12:16.800225 27195 tablet_bootstrap.cc:427] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:16.803643 27195 ts_tablet_manager.cc:1045] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.058s	user 0.007s	sys 0.003s
I0619 00:12:16.803372 27199 ts_tablet_manager.cc:1403] T 98df4155785241398c7270720d8a31cd P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:16.804394 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.805198 27195 raft_consensus.cc:325] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.805862 27195 raft_consensus.cc:351] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.806100 27195 raft_consensus.cc:628] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
W0619 00:12:16.806771 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.806737 27195 consensus_queue.cc:227] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.807847 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.810086 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.810770 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:16.811643 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.811887 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 80 ms (attempt = 2)
I0619 00:12:16.812173 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:16.812655 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98df4155785241398c7270720d8a31cd because the tablet was not found. No further retry: Not found: Tablet not found: 98df4155785241398c7270720d8a31cd
I0619 00:12:16.812916 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b1ca17b4629f4570bd99ddfbe12121fe" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:16.813040 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b1ca17b4629f4570bd99ddfbe12121fe" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:16.812964 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:16.813580 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.813545 27083 raft_consensus.cc:2732] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.897s (vote granted)
I0619 00:12:16.813946 27083 raft_consensus.cc:2245] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
W0619 00:12:16.814313 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:16.816015 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98df4155785241398c7270720d8a31cd because the tablet was not found. No further retry: Not found: Tablet not found: 98df4155785241398c7270720d8a31cd
W0619 00:12:16.816393 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.816921 26535 master_service.cc:523] ReplaceTablet: received request to replace tablet b1ca17b4629f4570bd99ddfbe12121fe from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:16.817729 26732 leader_election.cc:379] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:16.818049 26732 leader_election.cc:270] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:16.819181 27203 raft_consensus.cc:2732] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.836s (election complete)
I0619 00:12:16.819442 27203 raft_consensus.cc:2539] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:16.819519 27195 ts_tablet_manager.cc:1073] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
I0619 00:12:16.819705 27203 raft_consensus.cc:457] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:16.819885 27195 ts_tablet_manager.cc:1028] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:16.819917 27203 raft_consensus.cc:2732] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.589s (starting election)
I0619 00:12:16.820113 27203 raft_consensus.cc:2784] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:16.820206 27195 tablet_bootstrap.cc:427] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:16.823642 26536 master_service.cc:523] ReplaceTablet: received request to replace tablet 7485c30d9a814bf7b30223b93354bc22 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:16.824685 27195 tablet_bootstrap.cc:578] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:16.830205 26535 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet b1ca17b4629f4570bd99ddfbe12121fe
I0619 00:12:16.830168 27203 raft_consensus.cc:479] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.830929 26535 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1ca17b4629f4570bd99ddfbe12121fe on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:16.832103 26535 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1ca17b4629f4570bd99ddfbe12121fe on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:16.833464 27195 tablet_bootstrap.cc:427] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:16.833536 26535 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1ca17b4629f4570bd99ddfbe12121fe on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:16.833858 27195 ts_tablet_manager.cc:1045] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.005s
I0619 00:12:16.834272 26928 tablet_service.cc:800] Processing DeleteTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe with delete_type TABLET_DATA_DELETED (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.834442 27063 tablet_service.cc:800] Processing DeleteTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe with delete_type TABLET_DATA_DELETED (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:16.832381 26791 tablet_service.cc:800] Processing DeleteTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe with delete_type TABLET_DATA_DELETED (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:16.834960 27200 tablet_replica.cc:261] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:16.835211 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.835419 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.835561 26535 catalog_manager.cc:4596] ReplaceTablet: tablet b1ca17b4629f4570bd99ddfbe12121fe deleted and replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0
I0619 00:12:16.835619 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.836239 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.835882 27195 raft_consensus.cc:325] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.836453 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.836637 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.836608 27195 raft_consensus.cc:351] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.836835 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.836884 27195 raft_consensus.cc:628] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
W0619 00:12:16.837352 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet b1ca17b4629f4570bd99ddfbe12121fe with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b1ca17b4629f4570bd99ddfbe12121fe already in progress: creating tablet
I0619 00:12:16.837602 26521 catalog_manager.cc:2891] Scheduling retry of b1ca17b4629f4570bd99ddfbe12121fe Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 65 ms (attempt = 1)
I0619 00:12:16.837518 27195 consensus_queue.cc:227] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.840209 27199 tablet_replica.cc:261] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:16.840467 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.841104 27200 raft_consensus.cc:2042] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.842355 27200 raft_consensus.cc:2069] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:16.842694 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.842919 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.843098 27203 leader_election.cc:243] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:16.842504 27197 raft_consensus.cc:325] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.843397 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(b1ca17b4629f4570bd99ddfbe12121fe)
W0619 00:12:16.843739 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.843873 26536 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 7485c30d9a814bf7b30223b93354bc22
I0619 00:12:16.844177 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:16.844452 27197 raft_consensus.cc:351] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.844568 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:16.844563 27203 leader_election.cc:243] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:16.844866 27197 raft_consensus.cc:628] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:16.845561 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:16.845788 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:16.846194 26536 catalog_manager.cc:4596] ReplaceTablet: tablet 7485c30d9a814bf7b30223b93354bc22 deleted and replaced by tablet 8941e17c40ec418e9d3de771af3b96db
I0619 00:12:16.846212 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.2:46317
W0619 00:12:16.846820 26732 leader_election.cc:306] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:12:16.847265 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:16.847241 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 7485c30d9a814bf7b30223b93354bc22 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7485c30d9a814bf7b30223b93354bc22 already in progress: creating tablet
I0619 00:12:16.847463 26521 catalog_manager.cc:2891] Scheduling retry of 7485c30d9a814bf7b30223b93354bc22 Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 20 ms (attempt = 1)
I0619 00:12:16.847892 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:16.848404 27200 ts_tablet_manager.cc:1369] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:16.850744 26534 catalog_manager.cc:3663] Got report from deleted tablet 7485c30d9a814bf7b30223b93354bc22 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db): Sending delete request for this tablet
I0619 00:12:16.851457 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:16.846354 27197 consensus_queue.cc:227] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.854912 26791 tablet_service.cc:759] Processing CreateTablet for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.855191 27063 tablet_service.cc:759] Processing CreateTablet for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.855499 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.855482 26534 catalog_manager.cc:3663] Got report from deleted tablet 7485c30d9a814bf7b30223b93354bc22 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db): Sending delete request for this tablet
I0619 00:12:16.855780 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.856151 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:16.856403 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:16.861382 27176 flags.cc:406] Enabled experimental flag: --hive_metastore_sasl_enabled=false
I0619 00:12:16.861488 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:16.856616 26928 tablet_service.cc:759] Processing CreateTablet for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.864081 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:16.865052 27176 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:55188
W0619 00:12:16.866259 27176 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:12:16.866461 27176 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
I0619 00:12:16.866459 26929 tablet_service.cc:759] Processing CreateTablet for tablet 8941e17c40ec418e9d3de771af3b96db (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.866930 26792 tablet_service.cc:759] Processing CreateTablet for tablet 8941e17c40ec418e9d3de771af3b96db (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.867105 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8941e17c40ec418e9d3de771af3b96db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.867488 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8941e17c40ec418e9d3de771af3b96db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.868633 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:16.869335 27062 tablet_service.cc:759] Processing CreateTablet for tablet 8941e17c40ec418e9d3de771af3b96db (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.869988 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8941e17c40ec418e9d3de771af3b96db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.872787 26930 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.2:46317
W0619 00:12:16.866703 27176 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0619 00:12:16.873376 27176 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:12:16.873836 27176 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:12:16.874260 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b1ca17b4629f4570bd99ddfbe12121fe" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:16.874774 27083 raft_consensus.cc:2784] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:16.875066 27083 raft_consensus.cc:2732] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.763s (vote granted)
W0619 00:12:16.875170 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 7485c30d9a814bf7b30223b93354bc22 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7485c30d9a814bf7b30223b93354bc22 already in progress: creating tablet
I0619 00:12:16.875479 26521 catalog_manager.cc:2891] Scheduling retry of 7485c30d9a814bf7b30223b93354bc22 Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 43 ms (attempt = 2)
I0619 00:12:16.879176 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b1ca17b4629f4570bd99ddfbe12121fe" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
W0619 00:12:16.882059 26732 leader_election.cc:306] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:16.886317 27083 raft_consensus.cc:2245] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:16.888105 26928 ts_tablet_manager.cc:1173] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.888164 26732 leader_election.cc:379] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:16.888411 26732 leader_election.cc:270] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
W0619 00:12:16.888902 26732 raft_consensus.cc:2442] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:12:16.882972 26791 ts_tablet_manager.cc:1173] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.890082 27196 ts_tablet_manager.cc:1028] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:16.890108 26792 ts_tablet_manager.cc:1173] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.890465 27196 tablet_bootstrap.cc:427] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:16.892628 27196 tablet_bootstrap.cc:578] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:16.893087 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.894378 26930 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.2:46317
W0619 00:12:16.895365 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.895602 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 70 ms (attempt = 3)
I0619 00:12:16.894095 27200 ts_tablet_manager.cc:1382] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:16.896675 27200 log.cc:986] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/b1ca17b4629f4570bd99ddfbe12121fe
I0619 00:12:16.897420 27200 ts_tablet_manager.cc:1403] T b1ca17b4629f4570bd99ddfbe12121fe P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:16.898727 27062 ts_tablet_manager.cc:1173] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.900550 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet b1ca17b4629f4570bd99ddfbe12121fe (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:16.901249 27200 tablet_replica.cc:261] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:16.901860 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.902330 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.902967 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.903264 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1ca17b4629f4570bd99ddfbe12121fe on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:16.903401 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.903739 27063 ts_tablet_manager.cc:1173] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.904033 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.904481 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.904561 26928 tablet_service.cc:800] Processing DeleteTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe with delete_type TABLET_DATA_DELETED (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.904932 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.904947 26929 ts_tablet_manager.cc:1173] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.905357 27200 raft_consensus.cc:2042] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.905571 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(b1ca17b4629f4570bd99ddfbe12121fe)
W0619 00:12:16.906203 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet b1ca17b4629f4570bd99ddfbe12121fe with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b1ca17b4629f4570bd99ddfbe12121fe already in progress: creating tablet
I0619 00:12:16.906426 26521 catalog_manager.cc:2891] Scheduling retry of b1ca17b4629f4570bd99ddfbe12121fe Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 66 ms (attempt = 2)
I0619 00:12:16.906515 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.906723 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.906913 27199 raft_consensus.cc:2042] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.907187 27199 raft_consensus.cc:2069] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:16.906289 27195 ts_tablet_manager.cc:1073] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.072s	user 0.004s	sys 0.001s
I0619 00:12:16.907845 27195 ts_tablet_manager.cc:1028] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:16.908218 27195 tablet_bootstrap.cc:427] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:16.911089 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.909180 27199 ts_tablet_manager.cc:1369] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:16.911532 26534 catalog_manager.cc:3663] Got report from deleted tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe): Sending delete request for this tablet
I0619 00:12:16.912246 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.912581 27200 raft_consensus.cc:2069] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:16.913123 27195 tablet_bootstrap.cc:578] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
W0619 00:12:16.916424 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.917284 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.918247 26534 catalog_manager.cc:3663] Got report from deleted tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe): Sending delete request for this tablet
I0619 00:12:16.918920 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.919780 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:16.919915 27200 ts_tablet_manager.cc:1369] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:16.920187 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 98df4155785241398c7270720d8a31cd with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 98df4155785241398c7270720d8a31cd already in progress: creating tablet
I0619 00:12:16.920400 26521 catalog_manager.cc:2891] Scheduling retry of 98df4155785241398c7270720d8a31cd Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 19 ms (attempt = 1)
I0619 00:12:16.921540 27197 ts_tablet_manager.cc:1073] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.133s	user 0.014s	sys 0.011s
I0619 00:12:16.922030 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.922175 27195 tablet_bootstrap.cc:427] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:16.922276 27197 ts_tablet_manager.cc:1028] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:16.922554 27195 ts_tablet_manager.cc:1045] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.001s
I0619 00:12:16.922530 27210 tablet_replica.cc:261] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:16.922030 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.922638 27197 tablet_bootstrap.cc:427] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:16.922938 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.923171 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.923349 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.923528 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.923768 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.923943 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.924103 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(98df4155785241398c7270720d8a31cd)
I0619 00:12:16.924300 27210 raft_consensus.cc:2042] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.924592 27210 raft_consensus.cc:2069] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:16.926729 27210 ts_tablet_manager.cc:1369] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:16.927705 27196 tablet_bootstrap.cc:427] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:16.928040 27196 ts_tablet_manager.cc:1045] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.004s
I0619 00:12:16.928256 27197 tablet_bootstrap.cc:578] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:16.928396 27195 raft_consensus.cc:325] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.929482 27195 raft_consensus.cc:351] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.930011 27196 raft_consensus.cc:325] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.930683 27196 raft_consensus.cc:351] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.930732 27195 raft_consensus.cc:628] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:16.930938 27196 raft_consensus.cc:628] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:16.931385 27196 consensus_queue.cc:227] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.933316 27196 ts_tablet_manager.cc:1073] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:12:16.931459 27195 consensus_queue.cc:227] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.933710 27196 ts_tablet_manager.cc:1028] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
W0619 00:12:16.933918 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:16.935662 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:16.938513 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.940734 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.941550 27199 ts_tablet_manager.cc:1382] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:16.941906 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.945211 27195 ts_tablet_manager.cc:1073] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.001s
W0619 00:12:16.946720 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.947013 27195 ts_tablet_manager.cc:1028] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:16.947362 27195 tablet_bootstrap.cc:427] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:16.949651 27210 ts_tablet_manager.cc:1382] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:16.952404 27195 tablet_bootstrap.cc:578] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:16.951719 26540 rpcz_store.cc:265] Call kudu.master.MasterService.CreateTable from 127.24.119.62:33747 (request call id 3) took 1146ms. Request Metrics: {"hms-catalog.queue_time_us":834396,"hms-catalog.run_cpu_time_us":496,"hms-catalog.run_wall_time_us":283209,"child_traces":[["txn",{"apply.queue_time_us":79,"apply.run_cpu_time_us":1016,"apply.run_wall_time_us":1016,"num_ops":2,"prepare.queue_time_us":155,"prepare.run_cpu_time_us":1330,"prepare.run_wall_time_us":1331,"raft.queue_time_us":817,"raft.run_cpu_time_us":438,"raft.run_wall_time_us":437,"replication_time_us":5617}]]}
I0619 00:12:16.956547 27200 ts_tablet_manager.cc:1382] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:16.949931 27210 log.cc:986] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/98df4155785241398c7270720d8a31cd
I0619 00:12:16.958843 27210 ts_tablet_manager.cc:1403] T 98df4155785241398c7270720d8a31cd P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:16.961287 27197 tablet_bootstrap.cc:427] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:16.961647 27197 ts_tablet_manager.cc:1045] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.039s	user 0.004s	sys 0.003s
I0619 00:12:16.956836 27200 log.cc:986] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/7485c30d9a814bf7b30223b93354bc22
I0619 00:12:16.962086 26540 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:33747:
table { table_name: "default.table_f9c29cfbe84f43ff836467ab27c1a7ca" }
I0619 00:12:16.962528 27200 ts_tablet_manager.cc:1403] T 7485c30d9a814bf7b30223b93354bc22 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:16.963111 27197 raft_consensus.cc:325] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.965004 27197 raft_consensus.cc:351] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.965221 27197 raft_consensus.cc:628] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:16.966810 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98df4155785241398c7270720d8a31cd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:16.967223 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 7485c30d9a814bf7b30223b93354bc22 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:16.966245 27197 consensus_queue.cc:227] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.967857 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 98df4155785241398c7270720d8a31cd with delete_type TABLET_DATA_DELETED (replaced by tablet b1ca17b4629f4570bd99ddfbe12121fe) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.968350 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 98df4155785241398c7270720d8a31cd (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:16.969588 26929 tablet_service.cc:759] Processing CreateTablet for tablet df34b81bf336434c83017c49b5c6bc8b (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.969905 27195 tablet_bootstrap.cc:427] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:16.970182 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet df34b81bf336434c83017c49b5c6bc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.934053 27196 tablet_bootstrap.cc:427] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:16.970266 27195 ts_tablet_manager.cc:1045] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.023s	user 0.006s	sys 0.001s
W0619 00:12:16.970376 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 7485c30d9a814bf7b30223b93354bc22 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet 7485c30d9a814bf7b30223b93354bc22 already in progress: creating tablet
I0619 00:12:16.970603 26521 catalog_manager.cc:2891] Scheduling retry of 7485c30d9a814bf7b30223b93354bc22 Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 89 ms (attempt = 3)
W0619 00:12:16.972028 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 98df4155785241398c7270720d8a31cd because the tablet was not found. No further retry: Not found: Tablet not found: 98df4155785241398c7270720d8a31cd
I0619 00:12:16.971848 27195 raft_consensus.cc:325] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.972553 27195 raft_consensus.cc:351] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.972786 27195 raft_consensus.cc:628] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:16.973129 27196 tablet_bootstrap.cc:578] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:16.973598 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1ca17b4629f4570bd99ddfbe12121fe on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:16.973572 26792 tablet_service.cc:759] Processing CreateTablet for tablet df34b81bf336434c83017c49b5c6bc8b (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:16.973563 27062 tablet_service.cc:759] Processing CreateTablet for tablet df34b81bf336434c83017c49b5c6bc8b (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:16.973767 27138 tablet.cc:1678] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:12:16.974443 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet df34b81bf336434c83017c49b5c6bc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.974480 26928 tablet_service.cc:800] Processing DeleteTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe with delete_type TABLET_DATA_DELETED (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.975006 27199 log.cc:986] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/b1ca17b4629f4570bd99ddfbe12121fe
W0619 00:12:16.975486 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 98df4155785241398c7270720d8a31cd because the tablet was not found. No further retry: Not found: Tablet not found: 98df4155785241398c7270720d8a31cd
W0619 00:12:16.976524 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet b1ca17b4629f4570bd99ddfbe12121fe with error code TABLET_NOT_RUNNING: Already present: State transition of tablet b1ca17b4629f4570bd99ddfbe12121fe already in progress: creating tablet
I0619 00:12:16.976764 26521 catalog_manager.cc:2891] Scheduling retry of b1ca17b4629f4570bd99ddfbe12121fe Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 98 ms (attempt = 3)
I0619 00:12:16.977157 27199 ts_tablet_manager.cc:1403] T b1ca17b4629f4570bd99ddfbe12121fe P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:16.977442 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet df34b81bf336434c83017c49b5c6bc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:16.979320 27199 tablet_replica.cc:261] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
W0619 00:12:16.979737 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.979565 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.980427 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.980464 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet b1ca17b4629f4570bd99ddfbe12121fe (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:16.980669 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.980880 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.981111 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.981312 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.981508 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.981724 27199 raft_consensus.cc:2042] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.981904 26534 catalog_manager.cc:3663] Got report from deleted tablet 7485c30d9a814bf7b30223b93354bc22 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db): Sending delete request for this tablet
I0619 00:12:16.981997 27199 raft_consensus.cc:2069] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:16.982580 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
W0619 00:12:16.982749 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.983580 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.983873 27199 ts_tablet_manager.cc:1369] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:16.983873 27195 consensus_queue.cc:227] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:16.987350 26534 catalog_manager.cc:3663] Got report from deleted tablet 7485c30d9a814bf7b30223b93354bc22 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db): Sending delete request for this tablet
I0619 00:12:16.988447 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
W0619 00:12:16.988751 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.990592 27197 ts_tablet_manager.cc:1073] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.029s	user 0.003s	sys 0.001s
I0619 00:12:16.990950 27197 ts_tablet_manager.cc:1028] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:16.991322 27197 tablet_bootstrap.cc:427] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:16.991446 27062 ts_tablet_manager.cc:1173] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.993736 27197 tablet_bootstrap.cc:578] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:16.994273 27196 tablet_bootstrap.cc:427] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:16.994665 27196 ts_tablet_manager.cc:1045] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.061s	user 0.005s	sys 0.002s
I0619 00:12:16.996109 27196 raft_consensus.cc:325] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.997143 27196 raft_consensus.cc:351] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:16.997344 27196 raft_consensus.cc:628] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:16.997563 26792 ts_tablet_manager.cc:1173] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:16.994438 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:16.994871 27210 tablet_replica.cc:261] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:16.998042 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.998232 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.998404 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.998579 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:16.997817 27196 consensus_queue.cc:227] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:16.998795 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(7485c30d9a814bf7b30223b93354bc22)
W0619 00:12:17.001175 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.998968 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:17.001487 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(7485c30d9a814bf7b30223b93354bc22)
I0619 00:12:17.001652 27210 raft_consensus.cc:2042] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:16.995796 26929 ts_tablet_manager.cc:1173] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:16.992301 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:16.996124 27195 ts_tablet_manager.cc:1073] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.026s	user 0.004s	sys 0.000s
I0619 00:12:17.003366 27195 ts_tablet_manager.cc:1028] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.003774 27195 tablet_bootstrap.cc:427] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:17.005127 27210 raft_consensus.cc:2069] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.006434 27210 ts_tablet_manager.cc:1369] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.008219 27195 tablet_bootstrap.cc:578] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
W0619 00:12:17.008756 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.009856 27206 raft_consensus.cc:457] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.010109 27206 raft_consensus.cc:2732] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.887s (starting election)
I0619 00:12:17.010391 27206 raft_consensus.cc:479] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.011422 27206 leader_election.cc:243] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.011848 27206 leader_election.cc:243] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.013412 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8941e17c40ec418e9d3de771af3b96db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:17.013718 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8941e17c40ec418e9d3de771af3b96db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.014034 26812 raft_consensus.cc:2732] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.033s (vote granted)
I0619 00:12:17.014307 26812 raft_consensus.cc:2245] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
W0619 00:12:17.015313 27013 leader_election.cc:306] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:17.015635 27015 leader_election.cc:379] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.015861 27015 leader_election.cc:270] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.016286 27206 raft_consensus.cc:2732] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.506s (election complete)
I0619 00:12:17.016515 27206 raft_consensus.cc:2539] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.016731 27206 raft_consensus.cc:457] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.016916 27206 raft_consensus.cc:2732] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.050s (starting election)
I0619 00:12:17.017108 27206 raft_consensus.cc:2784] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.018254 27196 ts_tablet_manager.cc:1073] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.000s
I0619 00:12:17.018637 27196 ts_tablet_manager.cc:1028] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:17.018604 27197 tablet_bootstrap.cc:427] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.018946 27197 ts_tablet_manager.cc:1045] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.028s	user 0.005s	sys 0.001s
I0619 00:12:17.018980 27196 tablet_bootstrap.cc:427] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:17.022145 27196 tablet_bootstrap.cc:578] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:17.022058 27206 raft_consensus.cc:479] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.023097 27206 leader_election.cc:243] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.026641 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8941e17c40ec418e9d3de771af3b96db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
W0619 00:12:17.028605 27013 leader_election.cc:306] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:17.029228 27197 raft_consensus.cc:325] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.029882 27197 raft_consensus.cc:351] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.030160 27197 raft_consensus.cc:628] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.030409 27206 leader_election.cc:243] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.030720 27197 consensus_queue.cc:227] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.032094 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8941e17c40ec418e9d3de771af3b96db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:17.032557 26812 raft_consensus.cc:2784] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.032831 26812 raft_consensus.cc:2732] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.154s (vote granted)
I0619 00:12:17.032898 27199 ts_tablet_manager.cc:1382] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.033690 27195 tablet_bootstrap.cc:427] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:17.034126 27195 ts_tablet_manager.cc:1045] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.030s	user 0.004s	sys 0.001s
W0619 00:12:17.034592 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.035085 26536 catalog_manager.cc:3663] Got report from deleted tablet b1ca17b4629f4570bd99ddfbe12121fe (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0): Sending delete request for this tablet
I0619 00:12:17.035794 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1ca17b4629f4570bd99ddfbe12121fe on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0)
W0619 00:12:17.036182 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.036857 27210 ts_tablet_manager.cc:1382] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.037313 26929 tablet_service.cc:800] Processing DeleteTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe with delete_type TABLET_DATA_DELETED (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.037520 27176 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.61:43000
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.119.61
--webserver_port=40826
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:17.033344 27197 ts_tablet_manager.cc:1073] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I0619 00:12:17.038208 27197 ts_tablet_manager.cc:1028] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.040539 26812 raft_consensus.cc:2245] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:17.041280 26536 master_service.cc:523] ReplaceTablet: received request to replace tablet df34b81bf336434c83017c49b5c6bc8b from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.041764 27197 tablet_bootstrap.cc:427] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.041712 27015 leader_election.cc:379] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.041960 27015 leader_election.cc:270] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.043253 27206 raft_consensus.cc:2732] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.690s (election complete)
I0619 00:12:17.043524 27206 raft_consensus.cc:2539] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.044013 27197 tablet_bootstrap.cc:578] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.037124 27210 log.cc:986] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/7485c30d9a814bf7b30223b93354bc22
I0619 00:12:17.044333 27206 raft_consensus.cc:593] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:17.044917 27210 ts_tablet_manager.cc:1403] T 7485c30d9a814bf7b30223b93354bc22 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:17.048501 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 7485c30d9a814bf7b30223b93354bc22 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:17.048655 27210 tablet_replica.cc:261] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
W0619 00:12:17.048813 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.048995 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(b1ca17b4629f4570bd99ddfbe12121fe)
W0619 00:12:17.049252 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 7485c30d9a814bf7b30223b93354bc22 because the tablet was not found. No further retry: Not found: Tablet not found: 7485c30d9a814bf7b30223b93354bc22
I0619 00:12:17.049221 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:17.049705 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:17.037590 27195 raft_consensus.cc:325] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.049901 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:17.049976 27195 raft_consensus.cc:351] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.050109 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:17.050304 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:17.050500 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(b1ca17b4629f4570bd99ddfbe12121fe)
I0619 00:12:17.050633 27195 raft_consensus.cc:628] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:17.050729 27210 raft_consensus.cc:2042] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.049433 27206 consensus_queue.cc:204] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.052280 27210 raft_consensus.cc:2069] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.052702 26534 master_service.cc:523] ReplaceTablet: received request to replace tablet df34b81bf336434c83017c49b5c6bc8b from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.054177 27210 ts_tablet_manager.cc:1369] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.054838 26536 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet df34b81bf336434c83017c49b5c6bc8b
I0619 00:12:17.055136 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.055546 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.056092 26536 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.056257 26792 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.056418 26536 catalog_manager.cc:4596] ReplaceTablet: tablet df34b81bf336434c83017c49b5c6bc8b deleted and replaced by tablet acdbab1409d24027954bfbf5bc07ee3c
I0619 00:12:17.056603 26929 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.057196 27062 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:17.057673 27195 consensus_queue.cc:227] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.059703 27195 ts_tablet_manager.cc:1073] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.023s	user 0.004s	sys 0.000s
I0619 00:12:17.060093 27199 log.cc:986] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/7485c30d9a814bf7b30223b93354bc22
I0619 00:12:17.060398 27197 tablet_bootstrap.cc:427] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.060640 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7485c30d9a814bf7b30223b93354bc22 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.060765 27197 ts_tablet_manager.cc:1045] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.003s
W0619 00:12:17.061215 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.061662 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 7485c30d9a814bf7b30223b93354bc22 with delete_type TABLET_DATA_DELETED (replaced by tablet 8941e17c40ec418e9d3de771af3b96db) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.062608 27197 raft_consensus.cc:325] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:17.062991 26523 catalog_manager.cc:3083] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet df34b81bf336434c83017c49b5c6bc8b already in progress: creating tablet
I0619 00:12:17.063210 26523 catalog_manager.cc:2891] Scheduling retry of df34b81bf336434c83017c49b5c6bc8b Delete Tablet RPC for TS=10c4e713d4c44dd8904e5f77ba815785 with a delay of 44 ms (attempt = 1)
I0619 00:12:17.063253 27197 raft_consensus.cc:351] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.063307 27196 tablet_bootstrap.cc:427] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:17.063661 27196 ts_tablet_manager.cc:1045] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.045s	user 0.006s	sys 0.002s
W0619 00:12:17.063931 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.064425 27199 ts_tablet_manager.cc:1403] T 7485c30d9a814bf7b30223b93354bc22 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:17.063519 27197 raft_consensus.cc:628] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.066356 27197 consensus_queue.cc:227] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:17.069591 27197 ts_tablet_manager.cc:1073] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.001s
I0619 00:12:17.069965 27197 ts_tablet_manager.cc:1028] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.070323 27197 tablet_bootstrap.cc:427] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.068255 27196 raft_consensus.cc:325] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.071591 26792 tablet_service.cc:759] Processing CreateTablet for tablet acdbab1409d24027954bfbf5bc07ee3c (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.071625 27062 tablet_service.cc:759] Processing CreateTablet for tablet acdbab1409d24027954bfbf5bc07ee3c (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.071790 27196 raft_consensus.cc:351] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.072088 27196 raft_consensus.cc:628] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:17.072218 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet acdbab1409d24027954bfbf5bc07ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.072245 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet acdbab1409d24027954bfbf5bc07ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.072711 27196 consensus_queue.cc:227] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.073503 26929 tablet_service.cc:759] Processing CreateTablet for tablet acdbab1409d24027954bfbf5bc07ee3c (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.074090 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet acdbab1409d24027954bfbf5bc07ee3c. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:17.074853 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.076833 26534 catalog_manager.cc:3663] Got report from deleted tablet df34b81bf336434c83017c49b5c6bc8b (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c): Sending delete request for this tablet
I0619 00:12:17.077069 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1ca17b4629f4570bd99ddfbe12121fe on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.077468 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 7485c30d9a814bf7b30223b93354bc22 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:17.077544 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
W0619 00:12:17.078377 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.078419 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 7485c30d9a814bf7b30223b93354bc22 because the tablet was not found. No further retry: Not found: Tablet not found: 7485c30d9a814bf7b30223b93354bc22
W0619 00:12:17.078946 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 7485c30d9a814bf7b30223b93354bc22 because the tablet was not found. No further retry: Not found: Tablet not found: 7485c30d9a814bf7b30223b93354bc22
I0619 00:12:17.079391 26791 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.1:39548
W0619 00:12:17.079504 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.079804 26928 tablet_service.cc:800] Processing DeleteTablet for tablet b1ca17b4629f4570bd99ddfbe12121fe with delete_type TABLET_DATA_DELETED (replaced by tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0) from {username='jenkins'} at 127.24.119.2:46317
W0619 00:12:17.080782 26523 catalog_manager.cc:3083] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet df34b81bf336434c83017c49b5c6bc8b already in progress: creating tablet
I0619 00:12:17.081037 26523 catalog_manager.cc:2891] Scheduling retry of df34b81bf336434c83017c49b5c6bc8b Delete Tablet RPC for TS=10c4e713d4c44dd8904e5f77ba815785 with a delay of 63 ms (attempt = 1)
I0619 00:12:17.081840 27199 tablet_replica.cc:261] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:17.082671 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.083031 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.083129 26534 catalog_manager.cc:3663] Got report from deleted tablet df34b81bf336434c83017c49b5c6bc8b (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c): Sending delete request for this tablet
I0619 00:12:17.083256 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.083448 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.083685 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.083777 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.083873 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.085171 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(df34b81bf336434c83017c49b5c6bc8b)
W0619 00:12:17.085288 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.085536 27199 raft_consensus.cc:2042] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.086019 26791 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.086947 27199 raft_consensus.cc:2069] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:17.087002 26523 catalog_manager.cc:3083] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet df34b81bf336434c83017c49b5c6bc8b already in progress: creating tablet
I0619 00:12:17.087255 26523 catalog_manager.cc:2891] Scheduling retry of df34b81bf336434c83017c49b5c6bc8b Delete Tablet RPC for TS=10c4e713d4c44dd8904e5f77ba815785 with a delay of 29 ms (attempt = 1)
I0619 00:12:17.088912 27199 ts_tablet_manager.cc:1369] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.092203 27196 ts_tablet_manager.cc:1073] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.000s
I0619 00:12:17.093317 26929 ts_tablet_manager.cc:1173] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.097461 26792 ts_tablet_manager.cc:1173] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.098265 27210 ts_tablet_manager.cc:1382] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.098503 27210 log.cc:986] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/b1ca17b4629f4570bd99ddfbe12121fe
I0619 00:12:17.100380 27197 tablet_bootstrap.cc:578] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.102102 26534 catalog_manager.cc:3786] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:17.103597 27210 ts_tablet_manager.cc:1403] T b1ca17b4629f4570bd99ddfbe12121fe P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:17.103785 27176 master_main.cc:80] Initializing master server...
I0619 00:12:17.100378 27062 ts_tablet_manager.cc:1173] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.105943 27195 ts_tablet_manager.cc:1028] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.102100 27203 raft_consensus.cc:457] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.106237 27203 raft_consensus.cc:2732] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.067s (starting election)
I0619 00:12:17.106309 27195 tablet_bootstrap.cc:427] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:17.106520 27203 raft_consensus.cc:479] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.107587 27203 leader_election.cc:243] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.108033 27203 leader_election.cc:243] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.109187 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet b1ca17b4629f4570bd99ddfbe12121fe (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:17.109223 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df34b81bf336434c83017c49b5c6bc8b" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:17.109715 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet df34b81bf336434c83017c49b5c6bc8b already in progress: creating tablet
I0619 00:12:17.109798 27195 tablet_bootstrap.cc:578] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:17.109966 26521 catalog_manager.cc:2891] Scheduling retry of df34b81bf336434c83017c49b5c6bc8b Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 49 ms (attempt = 1)
I0619 00:12:17.110188 27176 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 456913us
I0619 00:12:17.110090 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df34b81bf336434c83017c49b5c6bc8b" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
W0619 00:12:17.110409 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 7485c30d9a814bf7b30223b93354bc22 because the tablet was not found. No further retry: Not found: Tablet not found: 7485c30d9a814bf7b30223b93354bc22
I0619 00:12:17.110882 27196 ts_tablet_manager.cc:1028] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:17.111261 27196 tablet_bootstrap.cc:427] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:17.111873 26732 leader_election.cc:306] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:12:17.112354 26732 leader_election.cc:306] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:12:17.112598 26732 leader_election.cc:270] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
W0619 00:12:17.112757 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet b1ca17b4629f4570bd99ddfbe12121fe because the tablet was not found. No further retry: Not found: Tablet not found: b1ca17b4629f4570bd99ddfbe12121fe
I0619 00:12:17.113032 27203 raft_consensus.cc:2732] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.027s (election complete)
I0619 00:12:17.113343 27199 ts_tablet_manager.cc:1382] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.113487 26523 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.113574 27199 log.cc:986] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/df34b81bf336434c83017c49b5c6bc8b
I0619 00:12:17.113243 27203 raft_consensus.cc:2487] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:17.114084 27199 ts_tablet_manager.cc:1403] T df34b81bf336434c83017c49b5c6bc8b P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:17.115326 26792 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.115833 27200 tablet_replica.cc:261] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:17.116066 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.116271 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.116406 27196 tablet_bootstrap.cc:578] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:17.117141 27176 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:17.116462 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.117388 27176 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:17.117429 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.117679 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.117882 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.118079 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.118263 27200 raft_consensus.cc:2042] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.118510 27197 tablet_bootstrap.cc:427] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.118531 27200 raft_consensus.cc:2069] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.118901 27197 ts_tablet_manager.cc:1045] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.049s	user 0.007s	sys 0.003s
I0619 00:12:17.119009 26523 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.120187 27197 raft_consensus.cc:325] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.121640 26792 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.122697 27200 ts_tablet_manager.cc:1369] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.124249 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet df34b81bf336434c83017c49b5c6bc8b (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:17.127136 27195 tablet_bootstrap.cc:427] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:17.127507 27195 ts_tablet_manager.cc:1045] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.002s
I0619 00:12:17.129565 27195 raft_consensus.cc:325] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.130239 27195 raft_consensus.cc:351] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.127830 27197 raft_consensus.cc:351] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.130600 27197 raft_consensus.cc:628] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.131127 27197 consensus_queue.cc:227] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.134589 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.135653 27197 ts_tablet_manager.cc:1073] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.000s
W0619 00:12:17.136153 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.136638 27197 ts_tablet_manager.cc:1028] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.130444 27195 raft_consensus.cc:628] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:17.138134 27195 consensus_queue.cc:227] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.138967 27206 raft_consensus.cc:457] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.139384 27206 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.754s (starting election)
I0619 00:12:17.140630 27206 raft_consensus.cc:479] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:17.140247 27196 tablet_bootstrap.cc:427] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:17.141358 27196 ts_tablet_manager.cc:1045] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.030s	user 0.004s	sys 0.002s
I0619 00:12:17.142763 27196 raft_consensus.cc:325] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.143400 27196 raft_consensus.cc:351] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.143635 27196 raft_consensus.cc:628] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:17.144863 27197 tablet_bootstrap.cc:427] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.144979 27176 fs_manager.cc:397] Time spent opening directory manager: real 0.021s	user 0.002s	sys 0.009s
I0619 00:12:17.146232 27176 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:17.147747 27176 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:17.148689 26523 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.147150 27196 consensus_queue.cc:227] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.149777 26792 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.151082 27200 ts_tablet_manager.cc:1382] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.151314 27200 log.cc:986] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/df34b81bf336434c83017c49b5c6bc8b
I0619 00:12:17.151839 27200 ts_tablet_manager.cc:1403] T df34b81bf336434c83017c49b5c6bc8b P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:17.154585 27197 tablet_bootstrap.cc:578] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.157413 27196 ts_tablet_manager.cc:1073] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
W0619 00:12:17.159102 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.159837 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.161114 26929 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.2:46317
W0619 00:12:17.162238 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.162241 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet df34b81bf336434c83017c49b5c6bc8b already in progress: creating tablet
I0619 00:12:17.162480 26521 catalog_manager.cc:2891] Scheduling retry of df34b81bf336434c83017c49b5c6bc8b Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 47 ms (attempt = 2)
W0619 00:12:17.162827 26865 tablet.cc:1678] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:12:17.163889 27197 tablet_bootstrap.cc:427] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.164206 27197 ts_tablet_manager.cc:1045] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.020s	user 0.005s	sys 0.000s
I0619 00:12:17.165771 27197 raft_consensus.cc:325] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.158247 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet df34b81bf336434c83017c49b5c6bc8b (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:17.166415 27197 raft_consensus.cc:351] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.166611 27197 raft_consensus.cc:628] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
W0619 00:12:17.167189 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b because the tablet was not found. No further retry: Not found: Tablet not found: df34b81bf336434c83017c49b5c6bc8b
I0619 00:12:17.167079 27197 consensus_queue.cc:227] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.143345 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.168496 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b because the tablet was not found. No further retry: Not found: Tablet not found: df34b81bf336434c83017c49b5c6bc8b
I0619 00:12:17.169324 27206 leader_election.cc:243] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:17.170665 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.171391 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.169656 27195 ts_tablet_manager.cc:1073] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.000s
I0619 00:12:17.172240 27206 leader_election.cc:243] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.172380 26534 catalog_manager.cc:3663] Got report from deleted tablet df34b81bf336434c83017c49b5c6bc8b (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c): Sending delete request for this tablet
I0619 00:12:17.172296 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5a63eb3f8e3e48cd9d0ceb3a81e901b0" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.172825 26950 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.985s (vote granted)
I0619 00:12:17.173096 27176 fs_manager.cc:417] Time spent opening block manager: real 0.014s	user 0.000s	sys 0.003s
I0619 00:12:17.173111 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.173084 26950 raft_consensus.cc:2245] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:17.174299 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5a63eb3f8e3e48cd9d0ceb3a81e901b0" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:17.174721 26929 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.174800 26812 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.541s (vote granted)
I0619 00:12:17.173274 27176 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
uuid: "b8582cfc277d4d4597c5605b7ac0ed48"
format_stamp: "Formatted at 2018-06-19 07:12:11 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:17.175060 26812 raft_consensus.cc:2245] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:17.175297 27176 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
W0619 00:12:17.175894 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet df34b81bf336434c83017c49b5c6bc8b already in progress: creating tablet
I0619 00:12:17.176125 26521 catalog_manager.cc:2891] Scheduling retry of df34b81bf336434c83017c49b5c6bc8b Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 26 ms (attempt = 1)
I0619 00:12:17.176159 27013 leader_election.cc:379] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.176393 27013 leader_election.cc:270] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.176882 27015 leader_election.cc:379] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:17.179034 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.179705 26534 catalog_manager.cc:3663] Got report from deleted tablet df34b81bf336434c83017c49b5c6bc8b (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c): Sending delete request for this tablet
I0619 00:12:17.180346 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.181565 27206 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.576s (election complete)
I0619 00:12:17.181843 27206 raft_consensus.cc:2539] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.181857 26929 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.182059 27206 raft_consensus.cc:457] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.182266 27206 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.841s (starting election)
I0619 00:12:17.182474 27206 raft_consensus.cc:2784] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.182772 27197 ts_tablet_manager.cc:1073] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
W0619 00:12:17.182845 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b with error code TABLET_NOT_RUNNING: Already present: State transition of tablet df34b81bf336434c83017c49b5c6bc8b already in progress: creating tablet
I0619 00:12:17.183058 26521 catalog_manager.cc:2891] Scheduling retry of df34b81bf336434c83017c49b5c6bc8b Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 47 ms (attempt = 1)
I0619 00:12:17.183173 27197 ts_tablet_manager.cc:1028] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.183560 27197 tablet_bootstrap.cc:427] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.187199 27197 tablet_bootstrap.cc:578] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.191123 27206 raft_consensus.cc:479] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:17.192167 27206 leader_election.cc:243] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:17.193758 27003 tablet.cc:1678] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0619 00:12:17.193868 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5a63eb3f8e3e48cd9d0ceb3a81e901b0" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:17.194344 26950 raft_consensus.cc:2784] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.194644 26950 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.162s (vote granted)
I0619 00:12:17.197682 27206 leader_election.cc:243] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.199828 26950 raft_consensus.cc:2245] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:17.200924 27013 leader_election.cc:379] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.201207 27013 leader_election.cc:270] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.201792 27204 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.512s (election complete)
I0619 00:12:17.202129 27204 raft_consensus.cc:2539] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.202714 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.203867 26929 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.204327 27210 tablet_replica.cc:261] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:17.204567 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.204815 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.205421 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.205595 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.205840 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.206037 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.206198 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(df34b81bf336434c83017c49b5c6bc8b)
I0619 00:12:17.206392 27210 raft_consensus.cc:2042] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.206668 27210 raft_consensus.cc:2069] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.206971 27197 tablet_bootstrap.cc:427] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.207321 27197 ts_tablet_manager.cc:1045] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.024s	user 0.006s	sys 0.004s
I0619 00:12:17.208847 27210 ts_tablet_manager.cc:1369] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.208667 27197 raft_consensus.cc:325] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.209369 27197 raft_consensus.cc:351] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.209559 27197 raft_consensus.cc:628] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.211400 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.212371 26929 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.213613 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5a63eb3f8e3e48cd9d0ceb3a81e901b0" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:17.214071 26812 raft_consensus.cc:2784] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.214326 26812 raft_consensus.cc:2732] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.816s (vote granted)
I0619 00:12:17.210098 27197 consensus_queue.cc:227] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.217977 26538 rpcz_store.cc:265] Call kudu.master.MasterService.CreateTable from 127.24.119.62:33747 (request call id 5) took 1395ms. Request Metrics: {"hms-catalog.queue_time_us":1115717,"hms-catalog.run_cpu_time_us":492,"hms-catalog.run_wall_time_us":264876,"child_traces":[["txn",{"apply.queue_time_us":64,"apply.run_cpu_time_us":993,"apply.run_wall_time_us":994,"num_ops":2,"prepare.queue_time_us":143,"prepare.run_cpu_time_us":1388,"prepare.run_wall_time_us":1387,"raft.queue_time_us":1019,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":502,"replication_time_us":5132}]]}
W0619 00:12:17.219157 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.222332 26812 raft_consensus.cc:2245] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:17.223518 27015 leader_election.cc:379] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:17.224838 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.224932 26538 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:33747:
table { table_name: "default.table_d9d85b384d834dd9ac9814f9d0fa9ce3" }
I0619 00:12:17.232684 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df34b81bf336434c83017c49b5c6bc8b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:17.232987 27197 ts_tablet_manager.cc:1073] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.025s	user 0.003s	sys 0.000s
I0619 00:12:17.233634 26929 tablet_service.cc:800] Processing DeleteTablet for tablet df34b81bf336434c83017c49b5c6bc8b with delete_type TABLET_DATA_DELETED (replaced by tablet acdbab1409d24027954bfbf5bc07ee3c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.233821 27203 raft_consensus.cc:457] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.234071 27203 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.022s (starting election)
W0619 00:12:17.234522 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.234323 27203 raft_consensus.cc:479] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.235337 27203 leader_election.cc:243] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.235766 27203 leader_election.cc:243] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.236932 27062 tablet_service.cc:759] Processing CreateTablet for tablet d36c61074b784b8a914abf3564cade33 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.237556 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d36c61074b784b8a914abf3564cade33. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.237830 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acdbab1409d24027954bfbf5bc07ee3c" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:17.238339 27083 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.157s (vote granted)
I0619 00:12:17.238565 27083 raft_consensus.cc:2245] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:17.239681 26732 leader_election.cc:379] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.239908 26732 leader_election.cc:270] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.239898 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acdbab1409d24027954bfbf5bc07ee3c" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.240809 26950 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.626s (vote granted)
I0619 00:12:17.241029 26950 raft_consensus.cc:2245] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:17.241968 27203 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.578s (election complete)
I0619 00:12:17.242058 26732 leader_election.cc:379] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.242195 27203 raft_consensus.cc:2539] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.242420 27203 raft_consensus.cc:457] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.242635 27203 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.843s (starting election)
I0619 00:12:17.242832 27203 raft_consensus.cc:2784] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.244177 26792 tablet_service.cc:759] Processing CreateTablet for tablet d36c61074b784b8a914abf3564cade33 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.244762 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d36c61074b784b8a914abf3564cade33. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.241185 26929 tablet_service.cc:759] Processing CreateTablet for tablet d36c61074b784b8a914abf3564cade33 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.245206 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d36c61074b784b8a914abf3564cade33. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.250571 27062 ts_tablet_manager.cc:1173] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.251051 27195 ts_tablet_manager.cc:1028] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.251402 27195 tablet_bootstrap.cc:427] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:17.253532 27195 tablet_bootstrap.cc:578] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:17.256552 27203 raft_consensus.cc:479] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.258136 27203 leader_election.cc:243] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.258368 26929 ts_tablet_manager.cc:1173] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.259433 26534 master_service.cc:523] ReplaceTablet: received request to replace tablet 8941e17c40ec418e9d3de771af3b96db from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.259717 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acdbab1409d24027954bfbf5bc07ee3c" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:17.260123 26950 raft_consensus.cc:2784] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.260345 26950 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.665s (vote granted)
I0619 00:12:17.260852 27204 raft_consensus.cc:593] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:17.261344 27204 consensus_queue.cc:204] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:17.265110 26950 raft_consensus.cc:2245] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:17.266443 26732 leader_election.cc:379] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.266695 26732 leader_election.cc:270] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.266778 27197 ts_tablet_manager.cc:1028] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.267153 27197 tablet_bootstrap.cc:427] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.268437 27210 ts_tablet_manager.cc:1382] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.269245 27256 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.238s (election complete)
I0619 00:12:17.269824 27203 leader_election.cc:243] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.270125 27210 log.cc:986] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/df34b81bf336434c83017c49b5c6bc8b
I0619 00:12:17.271453 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 8941e17c40ec418e9d3de771af3b96db from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.273682 26534 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 8941e17c40ec418e9d3de771af3b96db
I0619 00:12:17.273972 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8941e17c40ec418e9d3de771af3b96db on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet e5c538f18f88491787160e68800e29d7)
I0619 00:12:17.274377 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8941e17c40ec418e9d3de771af3b96db on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet e5c538f18f88491787160e68800e29d7)
I0619 00:12:17.275019 27210 ts_tablet_manager.cc:1403] T df34b81bf336434c83017c49b5c6bc8b P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:17.274765 26534 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8941e17c40ec418e9d3de771af3b96db on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet e5c538f18f88491787160e68800e29d7)
I0619 00:12:17.275496 26534 catalog_manager.cc:4596] ReplaceTablet: tablet 8941e17c40ec418e9d3de771af3b96db deleted and replaced by tablet e5c538f18f88491787160e68800e29d7
I0619 00:12:17.277200 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 8941e17c40ec418e9d3de771af3b96db with delete_type TABLET_DATA_DELETED (replaced by tablet e5c538f18f88491787160e68800e29d7) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:17.277695 27199 tablet_replica.cc:261] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:17.277915 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.278102 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.278272 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.278291 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 8941e17c40ec418e9d3de771af3b96db with delete_type TABLET_DATA_DELETED (replaced by tablet e5c538f18f88491787160e68800e29d7) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.278460 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.278607 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "acdbab1409d24027954bfbf5bc07ee3c" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:17.278688 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.279114 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.279314 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.279479 27199 raft_consensus.cc:2042] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:17.279688 27199 consensus_peers.cc:456] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:17.279892 27199 consensus_peers.cc:456] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:17.280238 27199 pending_rounds.cc:63] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:17.280421 27199 pending_rounds.cc:68] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391012571901952 op_type: NO_OP noop_request { }
I0619 00:12:17.280758 27199 raft_consensus.cc:2622] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:17.280966 27199 raft_consensus.cc:2069] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.281338 27197 tablet_bootstrap.cc:578] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.279119 27083 raft_consensus.cc:2784] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.283756 27083 raft_consensus.cc:2732] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.170s (vote granted)
I0619 00:12:17.284973 26539 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:33747:
table { table_name: "default.table_4a6e4e28e18547719a48db319fc95750" } new_table_name: "default.table_8a99507c6bce41038f15f6aea10b464e" alter_external_catalogs: true
I0619 00:12:17.290913 26535 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_06c34c9cf69d48a7be4493acace87da2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:17.291148 26536 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_3a31a472b2e1460591b55ec649f29582"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:17.291786 26535 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_06c34c9cf69d48a7be4493acace87da2 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:12:17.292018 26536 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3a31a472b2e1460591b55ec649f29582 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:17.290851 26537 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_f4f93d7c11cf4735980b09abd2aa7f62"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:17.293778 26537 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f4f93d7c11cf4735980b09abd2aa7f62 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:17.294118 26541 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:33747:
table { table_name: "default.table_31d388c40f064a2a97bbbec12c77fbd9" } new_table_name: "default.table_f3e7e07d4ed74a6cac218172903cccfd" alter_external_catalogs: true
I0619 00:12:17.295516 27083 raft_consensus.cc:2245] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:17.295469 26790 tablet_service.cc:759] Processing CreateTablet for tablet e5c538f18f88491787160e68800e29d7 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.296581 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5c538f18f88491787160e68800e29d7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.296708 26929 tablet_service.cc:759] Processing CreateTablet for tablet e5c538f18f88491787160e68800e29d7 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.296900 26732 leader_election.cc:379] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.297719 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5c538f18f88491787160e68800e29d7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.276280 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 8941e17c40ec418e9d3de771af3b96db with delete_type TABLET_DATA_DELETED (replaced by tablet e5c538f18f88491787160e68800e29d7) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.269518 27256 raft_consensus.cc:2539] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.299141 27200 tablet_replica.cc:261] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:17.299388 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.299589 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.299793 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.299964 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.300156 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.300335 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.300503 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.300498 27062 tablet_service.cc:759] Processing CreateTablet for tablet e5c538f18f88491787160e68800e29d7 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.300698 27200 raft_consensus.cc:2042] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.300948 27200 raft_consensus.cc:2069] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.301234 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e5c538f18f88491787160e68800e29d7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.302254 27200 ts_tablet_manager.cc:1369] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.306757 26929 ts_tablet_manager.cc:1173] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.309154 26792 ts_tablet_manager.cc:1173] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.309609 27196 ts_tablet_manager.cc:1028] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:17.309965 27196 tablet_bootstrap.cc:427] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:17.310498 27199 ts_tablet_manager.cc:1369] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.310462 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet df34b81bf336434c83017c49b5c6bc8b (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:17.313012 27195 tablet_bootstrap.cc:427] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:17.313308 27195 ts_tablet_manager.cc:1045] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.062s	user 0.004s	sys 0.005s
I0619 00:12:17.317596 27200 ts_tablet_manager.cc:1382] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.317883 27200 log.cc:986] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/8941e17c40ec418e9d3de771af3b96db
I0619 00:12:17.318356 27200 ts_tablet_manager.cc:1403] T 8941e17c40ec418e9d3de771af3b96db P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:17.320149 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 8941e17c40ec418e9d3de771af3b96db (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:17.328778 26790 ts_tablet_manager.cc:1173] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.329998 27196 tablet_bootstrap.cc:578] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:17.311444 27197 tablet_bootstrap.cc:427] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.330399 27197 ts_tablet_manager.cc:1045] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.063s	user 0.005s	sys 0.001s
I0619 00:12:17.331868 27197 raft_consensus.cc:325] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.332450 27197 raft_consensus.cc:351] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.332619 27197 raft_consensus.cc:628] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
W0619 00:12:17.330966 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.333570 27197 consensus_queue.cc:227] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.335513 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b because the tablet was not found. No further retry: Not found: Tablet not found: df34b81bf336434c83017c49b5c6bc8b
W0619 00:12:17.335995 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet df34b81bf336434c83017c49b5c6bc8b because the tablet was not found. No further retry: Not found: Tablet not found: df34b81bf336434c83017c49b5c6bc8b
I0619 00:12:17.336066 27195 raft_consensus.cc:325] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.337260 27195 raft_consensus.cc:351] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.337468 27195 raft_consensus.cc:628] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:17.338028 27195 consensus_queue.cc:227] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.339121 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.340469 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.340790 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.341629 27197 ts_tablet_manager.cc:1073] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.000s
I0619 00:12:17.334895 27210 tablet_replica.cc:261] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:17.341950 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.342150 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.342348 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.342537 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(8941e17c40ec418e9d3de771af3b96db)
W0619 00:12:17.343111 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.342767 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.343646 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.343884 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(8941e17c40ec418e9d3de771af3b96db)
I0619 00:12:17.344189 27199 ts_tablet_manager.cc:1382] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:17.344619 27210 raft_consensus.cc:2042] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.344938 27210 raft_consensus.cc:2069] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.345433 27195 ts_tablet_manager.cc:1073] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.003s
I0619 00:12:17.315754 27256 raft_consensus.cc:593] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:17.347275 27210 ts_tablet_manager.cc:1369] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.347009 27256 consensus_queue.cc:204] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.351967 27196 tablet_bootstrap.cc:427] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:17.352437 27196 ts_tablet_manager.cc:1045] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.043s	user 0.007s	sys 0.000s
I0619 00:12:17.352839 27197 ts_tablet_manager.cc:1028] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.353217 27197 tablet_bootstrap.cc:427] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.353788 27196 raft_consensus.cc:325] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.354689 27196 raft_consensus.cc:351] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.355728 27197 tablet_bootstrap.cc:578] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.356645 27196 raft_consensus.cc:628] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:17.357715 27196 consensus_queue.cc:227] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.360769 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.362267 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.363301 27062 ts_tablet_manager.cc:1173] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.363713 27195 ts_tablet_manager.cc:1028] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.364064 27195 tablet_bootstrap.cc:427] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:17.365037 27197 tablet_bootstrap.cc:427] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.364163 27196 ts_tablet_manager.cc:1073] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.000s
I0619 00:12:17.365833 27197 ts_tablet_manager.cc:1045] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.000s
I0619 00:12:17.365947 27196 ts_tablet_manager.cc:1028] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:17.366212 27195 tablet_bootstrap.cc:578] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:17.366710 27196 tablet_bootstrap.cc:427] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:17.367456 27197 raft_consensus.cc:325] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.370158 27196 tablet_bootstrap.cc:578] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:17.374320 27197 raft_consensus.cc:351] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.374523 27197 raft_consensus.cc:628] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.375010 27197 consensus_queue.cc:227] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.376344 26534 catalog_manager.cc:3786] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } }
W0619 00:12:17.389508 26524 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000): Couldn't send request to peer b8582cfc277d4d4597c5605b7ac0ed48 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times.
I0619 00:12:17.344446 27199 log.cc:986] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/8941e17c40ec418e9d3de771af3b96db
I0619 00:12:17.391466 27199 ts_tablet_manager.cc:1403] T 8941e17c40ec418e9d3de771af3b96db P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:17.393849 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 8941e17c40ec418e9d3de771af3b96db (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:17.394560 27222 raft_consensus.cc:457] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.394852 27222 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.950s (starting election)
I0619 00:12:17.395113 27222 raft_consensus.cc:479] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.397300 27222 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:17.398300 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.400328 26534 catalog_manager.cc:3786] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
W0619 00:12:17.402073 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.404604 27222 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.416811 27197 ts_tablet_manager.cc:1073] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.050s	user 0.004s	sys 0.000s
I0619 00:12:17.371492 27210 ts_tablet_manager.cc:1382] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.418146 27210 log.cc:986] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/8941e17c40ec418e9d3de771af3b96db
I0619 00:12:17.418722 27210 ts_tablet_manager.cc:1403] T 8941e17c40ec418e9d3de771af3b96db P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:17.420925 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 8941e17c40ec418e9d3de771af3b96db (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:17.425937 27196 tablet_bootstrap.cc:427] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:17.426246 27196 ts_tablet_manager.cc:1045] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.060s	user 0.005s	sys 0.001s
I0619 00:12:17.427510 27196 raft_consensus.cc:325] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.428130 27196 raft_consensus.cc:351] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.428879 27196 raft_consensus.cc:628] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:17.429338 27196 consensus_queue.cc:227] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.432117 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.433948 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.434499 27195 tablet_bootstrap.cc:427] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:17.435168 27195 ts_tablet_manager.cc:1045] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.071s	user 0.006s	sys 0.002s
I0619 00:12:17.435570 27196 ts_tablet_manager.cc:1073] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.001s
I0619 00:12:17.436556 27195 raft_consensus.cc:325] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.438181 27195 raft_consensus.cc:351] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.438385 27195 raft_consensus.cc:628] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:17.438853 27195 consensus_queue.cc:227] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.443804 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.449373 27195 ts_tablet_manager.cc:1073] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I0619 00:12:17.452492 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:17.452962 26812 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.112s (vote granted)
I0619 00:12:17.453173 26812 raft_consensus.cc:2245] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
W0619 00:12:17.449877 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.456296 26876 leader_election.cc:379] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.456557 26876 leader_election.cc:270] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.457159 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:17.457650 27083 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.011s (vote granted)
I0619 00:12:17.457937 27083 raft_consensus.cc:2245] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:17.458429 27222 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.665s (election complete)
I0619 00:12:17.458608 27204 raft_consensus.cc:457] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.458667 27222 raft_consensus.cc:2539] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.458819 27204 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.584s (starting election)
I0619 00:12:17.458904 27222 raft_consensus.cc:457] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.459125 27222 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.169s (starting election)
I0619 00:12:17.459328 27222 raft_consensus.cc:2784] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.459038 27204 raft_consensus.cc:479] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.460328 27204 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.460731 27204 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.462204 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.463580 26874 leader_election.cc:379] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.463557 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:17.464043 26812 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.066s (vote granted)
I0619 00:12:17.464267 26812 raft_consensus.cc:2245] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:17.466965 27222 raft_consensus.cc:479] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.467954 27222 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.468142 26950 raft_consensus.cc:2175] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:17.468530 27015 leader_election.cc:379] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.468773 27015 leader_election.cc:270] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.469221 27204 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.841s (election complete)
I0619 00:12:17.469444 27204 raft_consensus.cc:2539] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.469691 27204 raft_consensus.cc:457] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.469894 27204 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.618s (starting election)
I0619 00:12:17.470096 27204 raft_consensus.cc:2784] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.473065 27013 leader_election.cc:393] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Invalid argument: T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:17.473953 27222 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.475495 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:17.475952 26812 raft_consensus.cc:2784] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.476202 26812 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.643s (vote granted)
I0619 00:12:17.476032 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:17.475711 27204 raft_consensus.cc:479] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.479035 27083 raft_consensus.cc:2175] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 in current term 1: Already voted for candidate a2c1dfdaa6d442539e4342d184d2167c in this term.
I0619 00:12:17.480520 26812 raft_consensus.cc:2245] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:17.478366 27204 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.482024 26874 leader_election.cc:393] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote denied by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806). Message: Invalid argument: T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 in current term 1: Already voted for candidate a2c1dfdaa6d442539e4342d184d2167c in this term.
I0619 00:12:17.482507 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet d36c61074b784b8a914abf3564cade33 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.482672 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet d36c61074b784b8a914abf3564cade33 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.483203 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:17.483781 26950 raft_consensus.cc:2175] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:17.485134 27013 leader_election.cc:393] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Invalid argument: T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:17.482755 27204 leader_election.cc:243] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.489879 26876 leader_election.cc:379] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.490113 26876 leader_election.cc:270] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.490070 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d36c61074b784b8a914abf3564cade33" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:17.490644 26812 raft_consensus.cc:2175] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:17.490876 27222 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.078s (election complete)
I0619 00:12:17.491108 27222 raft_consensus.cc:2539] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.493216 27015 leader_election.cc:393] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote denied by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295). Message: Invalid argument: T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:17.493461 27015 leader_election.cc:270] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:12:17.494771 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet d36c61074b784b8a914abf3564cade33
I0619 00:12:17.495128 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d36c61074b784b8a914abf3564cade33 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 76754c71d69d462a83d462412180fb60)
I0619 00:12:17.495512 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d36c61074b784b8a914abf3564cade33 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 76754c71d69d462a83d462412180fb60)
I0619 00:12:17.496227 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d36c61074b784b8a914abf3564cade33 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 76754c71d69d462a83d462412180fb60)
I0619 00:12:17.496294 26929 tablet_service.cc:800] Processing DeleteTablet for tablet d36c61074b784b8a914abf3564cade33 with delete_type TABLET_DATA_DELETED (replaced by tablet 76754c71d69d462a83d462412180fb60) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.496868 26790 tablet_service.cc:800] Processing DeleteTablet for tablet d36c61074b784b8a914abf3564cade33 with delete_type TABLET_DATA_DELETED (replaced by tablet 76754c71d69d462a83d462412180fb60) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.496938 27210 tablet_replica.cc:261] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:17.496616 27204 raft_consensus.cc:2732] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.242s (election complete)
I0619 00:12:17.497169 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.497373 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.497344 27204 raft_consensus.cc:2487] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:12:17.497570 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.497799 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.498033 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.498230 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.498420 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.498888 27200 tablet_replica.cc:261] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:17.499145 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.499351 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.499542 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.499779 26533 catalog_manager.cc:4596] ReplaceTablet: tablet d36c61074b784b8a914abf3564cade33 deleted and replaced by tablet 76754c71d69d462a83d462412180fb60
I0619 00:12:17.499778 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.500007 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.500205 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.500394 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.500576 27200 raft_consensus.cc:2042] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.500860 27200 raft_consensus.cc:2069] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.502007 27062 tablet_service.cc:800] Processing DeleteTablet for tablet d36c61074b784b8a914abf3564cade33 with delete_type TABLET_DATA_DELETED (replaced by tablet 76754c71d69d462a83d462412180fb60) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:17.502486 27199 tablet_replica.cc:261] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:17.502733 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.502915 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.503079 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.503248 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.503444 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.503645 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.503813 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(d36c61074b784b8a914abf3564cade33)
I0619 00:12:17.503994 27199 raft_consensus.cc:2042] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.504245 27199 raft_consensus.cc:2069] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.505878 27200 ts_tablet_manager.cc:1369] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.506129 27199 ts_tablet_manager.cc:1369] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.512703 26929 tablet_service.cc:759] Processing CreateTablet for tablet 76754c71d69d462a83d462412180fb60 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.513160 27062 tablet_service.cc:759] Processing CreateTablet for tablet 76754c71d69d462a83d462412180fb60 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.513309 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 76754c71d69d462a83d462412180fb60. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.513736 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 76754c71d69d462a83d462412180fb60. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.517007 26790 tablet_service.cc:759] Processing CreateTablet for tablet 76754c71d69d462a83d462412180fb60 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.516348 27222 raft_consensus.cc:593] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:17.519431 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 76754c71d69d462a83d462412180fb60. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.519706 27222 consensus_queue.cc:204] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.522889 26929 ts_tablet_manager.cc:1173] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.523887 27200 ts_tablet_manager.cc:1382] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.527948 27200 log.cc:986] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/d36c61074b784b8a914abf3564cade33
I0619 00:12:17.528928 27200 ts_tablet_manager.cc:1403] T d36c61074b784b8a914abf3564cade33 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
W0619 00:12:17.525648 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.529537 27197 ts_tablet_manager.cc:1028] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.529932 27197 tablet_bootstrap.cc:427] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.531787 26533 catalog_manager.cc:3663] Got report from deleted tablet d36c61074b784b8a914abf3564cade33 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) (replaced by tablet 76754c71d69d462a83d462412180fb60): Sending delete request for this tablet
I0619 00:12:17.532181 27197 tablet_bootstrap.cc:578] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.532449 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet d36c61074b784b8a914abf3564cade33 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 76754c71d69d462a83d462412180fb60)
I0619 00:12:17.532639 26790 ts_tablet_manager.cc:1173] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.533275 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet d36c61074b784b8a914abf3564cade33 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:17.533571 27196 ts_tablet_manager.cc:1028] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:17.533633 27210 raft_consensus.cc:2042] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:17.533947 27196 tablet_bootstrap.cc:427] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:17.534272 27210 consensus_peers.cc:456] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:17.534495 27210 consensus_peers.cc:456] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:17.536936 26928 tablet_service.cc:800] Processing DeleteTablet for tablet d36c61074b784b8a914abf3564cade33 with delete_type TABLET_DATA_DELETED (replaced by tablet 76754c71d69d462a83d462412180fb60) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.537147 27210 pending_rounds.cc:63] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:17.537302 27210 pending_rounds.cc:68] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391014491443200 op_type: NO_OP noop_request { }
I0619 00:12:17.537652 27210 raft_consensus.cc:2622] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:17.537765 27196 tablet_bootstrap.cc:578] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:17.537904 27210 raft_consensus.cc:2069] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.541821 27204 raft_consensus.cc:457] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.542073 27204 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.969s (starting election)
I0619 00:12:17.542310 27204 raft_consensus.cc:479] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.541261 27271 consensus_queue.cc:1042] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1f719e8be9c9447a9acd40728c0a5710" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I0619 00:12:17.543284 27204 leader_election.cc:243] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.544836 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e5c538f18f88491787160e68800e29d7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:17.545292 26812 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.598s (vote granted)
I0619 00:12:17.545500 26812 raft_consensus.cc:2245] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:17.546039 27204 leader_election.cc:243] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.546958 27015 leader_election.cc:379] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.547583 27015 leader_election.cc:270] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.548169 27204 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.919s (election complete)
I0619 00:12:17.548420 27204 raft_consensus.cc:2539] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.548673 27204 raft_consensus.cc:457] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.548416 27062 ts_tablet_manager.cc:1173] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.548697 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e5c538f18f88491787160e68800e29d7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.549008 27204 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.013s (starting election)
I0619 00:12:17.549167 26950 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.078s (vote granted)
I0619 00:12:17.549288 27204 raft_consensus.cc:2784] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.549433 27195 ts_tablet_manager.cc:1028] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.549401 26950 raft_consensus.cc:2245] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:17.549836 27195 tablet_bootstrap.cc:427] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:17.550261 27197 tablet_bootstrap.cc:427] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.550585 27197 ts_tablet_manager.cc:1045] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.002s
I0619 00:12:17.550597 27013 leader_election.cc:379] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.552198 27195 tablet_bootstrap.cc:578] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:17.551972 27197 raft_consensus.cc:325] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.553555 27197 raft_consensus.cc:351] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.553814 27197 raft_consensus.cc:628] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.554561 27210 ts_tablet_manager.cc:1369] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.554255 27197 consensus_queue.cc:227] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.556340 27197 ts_tablet_manager.cc:1073] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
W0619 00:12:17.556646 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.560374 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.562057 27199 ts_tablet_manager.cc:1382] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.562309 27199 log.cc:986] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/d36c61074b784b8a914abf3564cade33
I0619 00:12:17.563347 27196 tablet_bootstrap.cc:427] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:17.563719 27196 ts_tablet_manager.cc:1045] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.030s	user 0.006s	sys 0.003s
I0619 00:12:17.565106 27196 raft_consensus.cc:325] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.565718 27196 raft_consensus.cc:351] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.565902 27196 raft_consensus.cc:628] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:17.566321 27196 consensus_queue.cc:227] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.568698 27195 tablet_bootstrap.cc:427] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:17.569022 27195 ts_tablet_manager.cc:1045] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.001s
I0619 00:12:17.569025 27204 raft_consensus.cc:479] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.569984 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.570330 27199 ts_tablet_manager.cc:1403] T d36c61074b784b8a914abf3564cade33 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:17.570407 27195 raft_consensus.cc:325] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.571023 27195 raft_consensus.cc:351] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.571210 27195 raft_consensus.cc:628] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:17.571651 27195 consensus_queue.cc:227] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.572670 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet d36c61074b784b8a914abf3564cade33 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:17.573557 27204 leader_election.cc:243] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:17.574211 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.574223 27204 leader_election.cc:243] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:17.575973 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.575871 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e5c538f18f88491787160e68800e29d7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:17.573443 27195 ts_tablet_manager.cc:1073] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:12:17.576314 26812 raft_consensus.cc:2784] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.576200 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e5c538f18f88491787160e68800e29d7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:17.576427 27196 ts_tablet_manager.cc:1073] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I0619 00:12:17.576678 26950 raft_consensus.cc:2784] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.576927 26950 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.624s (vote granted)
I0619 00:12:17.577603 26812 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.927s (vote granted)
I0619 00:12:17.580581 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:17.581809 27252 raft_consensus.cc:457] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.582067 27252 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.691s (starting election)
I0619 00:12:17.582348 27252 raft_consensus.cc:479] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.583367 27252 leader_election.cc:243] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.583382 26812 raft_consensus.cc:2245] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:17.583803 27252 leader_election.cc:243] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.586308 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "76754c71d69d462a83d462412180fb60" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.586833 27015 leader_election.cc:379] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.586858 26949 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.713s (vote granted)
I0619 00:12:17.587059 27015 leader_election.cc:270] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.587087 26949 raft_consensus.cc:2245] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:17.588765 27013 leader_election.cc:379] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.589004 27013 leader_election.cc:270] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.588850 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "76754c71d69d462a83d462412180fb60" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:17.589280 26812 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.125s (vote granted)
I0619 00:12:17.589426 27206 raft_consensus.cc:2732] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.117s (election complete)
I0619 00:12:17.589498 26812 raft_consensus.cc:2245] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:17.589681 27206 raft_consensus.cc:2539] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.590564 27015 leader_election.cc:379] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.591056 27252 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.034s (election complete)
I0619 00:12:17.591295 27252 raft_consensus.cc:2539] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.591513 27252 raft_consensus.cc:457] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.591774 27252 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.873s (starting election)
I0619 00:12:17.592011 27252 raft_consensus.cc:2784] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.597728 26950 raft_consensus.cc:2245] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:17.598373 27206 raft_consensus.cc:593] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:17.598935 27206 consensus_queue.cc:204] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.600253 27013 leader_election.cc:379] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:17.601977 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.602602 27210 ts_tablet_manager.cc:1382] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:17.602891 27210 log.cc:986] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/d36c61074b784b8a914abf3564cade33
I0619 00:12:17.603579 27210 ts_tablet_manager.cc:1403] T d36c61074b784b8a914abf3564cade33 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:17.603456 27252 raft_consensus.cc:479] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.604418 27252 leader_election.cc:243] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.604831 27252 leader_election.cc:243] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.606007 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "76754c71d69d462a83d462412180fb60" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:17.606452 26812 raft_consensus.cc:2784] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.606716 26812 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.659s (vote granted)
I0619 00:12:17.606978 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "76754c71d69d462a83d462412180fb60" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:17.607394 26950 raft_consensus.cc:2784] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.607640 26950 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.527s (vote granted)
I0619 00:12:17.611752 26950 raft_consensus.cc:2245] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:17.612160 26812 raft_consensus.cc:2245] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:17.613270 27015 leader_election.cc:379] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:17.613499 27015 leader_election.cc:270] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.613755 27013 leader_election.cc:379] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.614230 27252 raft_consensus.cc:2732] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.115s (election complete)
I0619 00:12:17.614853 27252 raft_consensus.cc:2539] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.615406 27252 raft_consensus.cc:593] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:17.616252 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet d36c61074b784b8a914abf3564cade33 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
W0619 00:12:17.617866 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet d36c61074b784b8a914abf3564cade33 because the tablet was not found. No further retry: Not found: Tablet not found: d36c61074b784b8a914abf3564cade33
I0619 00:12:17.616798 27252 consensus_queue.cc:204] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.618479 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.627418 26533 catalog_manager.cc:3786] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:17.653053 26533 catalog_manager.cc:3786] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:17.700531 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 76754c71d69d462a83d462412180fb60 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.708945 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.714480 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 76754c71d69d462a83d462412180fb60
I0619 00:12:17.714814 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 76754c71d69d462a83d462412180fb60 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.715168 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 76754c71d69d462a83d462412180fb60 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.715967 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 76754c71d69d462a83d462412180fb60 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.717346 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 76754c71d69d462a83d462412180fb60 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aff82640ec542be9aad6a4e782e45a8) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:17.717702 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 76754c71d69d462a83d462412180fb60 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aff82640ec542be9aad6a4e782e45a8) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.717851 27199 tablet_replica.cc:261] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:17.718125 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.718185 27210 tablet_replica.cc:261] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:17.718330 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.718425 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.718516 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.718628 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.718730 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.718840 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.718952 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.719033 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.719153 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.719251 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.719347 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.719439 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.719527 27199 raft_consensus.cc:2042] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:17.719622 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.719774 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 76754c71d69d462a83d462412180fb60 deleted and replaced by tablet 7aff82640ec542be9aad6a4e782e45a8
I0619 00:12:17.719763 27199 consensus_peers.cc:456] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:17.719825 27210 raft_consensus.cc:2042] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.719969 27199 consensus_peers.cc:456] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:17.720095 27210 raft_consensus.cc:2069] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.720324 27199 pending_rounds.cc:63] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:17.720479 27199 pending_rounds.cc:68] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391014888697856 op_type: NO_OP noop_request { }
I0619 00:12:17.720804 27199 raft_consensus.cc:2622] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:17.721029 27199 raft_consensus.cc:2069] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.722415 27210 ts_tablet_manager.cc:1369] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.723865 26790 tablet_service.cc:800] Processing DeleteTablet for tablet 76754c71d69d462a83d462412180fb60 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aff82640ec542be9aad6a4e782e45a8) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.724351 27200 tablet_replica.cc:261] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:17.724603 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.724817 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.725010 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.725185 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.725394 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.725600 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.725781 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(76754c71d69d462a83d462412180fb60)
I0619 00:12:17.725956 27200 raft_consensus.cc:2042] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.726215 27200 raft_consensus.cc:2069] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.727840 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0
I0619 00:12:17.728158 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:17.728277 27200 ts_tablet_manager.cc:1369] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.728564 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:17.728963 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:17.729208 27199 ts_tablet_manager.cc:1369] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.729274 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 deleted and replaced by tablet 2b4d721d47104c6a9f7e101a4bd19062
I0619 00:12:17.730474 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b4d721d47104c6a9f7e101a4bd19062) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.730641 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b4d721d47104c6a9f7e101a4bd19062) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:17.731487 26790 tablet_service.cc:800] Processing DeleteTablet for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b4d721d47104c6a9f7e101a4bd19062) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.737758 26790 tablet_service.cc:759] Processing CreateTablet for tablet 7aff82640ec542be9aad6a4e782e45a8 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.738355 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7aff82640ec542be9aad6a4e782e45a8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.739820 26929 tablet_service.cc:759] Processing CreateTablet for tablet 7aff82640ec542be9aad6a4e782e45a8 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.740402 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7aff82640ec542be9aad6a4e782e45a8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.740996 26950 raft_consensus.cc:1081] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:17.741039 27062 tablet_service.cc:759] Processing CreateTablet for tablet 7aff82640ec542be9aad6a4e782e45a8 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.741626 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7aff82640ec542be9aad6a4e782e45a8. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.744333 27256 consensus_queue.cc:963] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:17.749439 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.750033 26929 ts_tablet_manager.cc:1173] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.750442 27197 ts_tablet_manager.cc:1028] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.750798 27197 tablet_bootstrap.cc:427] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.753083 27197 tablet_bootstrap.cc:578] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.754927 26790 ts_tablet_manager.cc:1173] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.758679 26928 tablet_service.cc:759] Processing CreateTablet for tablet 2b4d721d47104c6a9f7e101a4bd19062 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.758847 27196 ts_tablet_manager.cc:1028] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:17.759183 27196 tablet_bootstrap.cc:427] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:17.759285 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b4d721d47104c6a9f7e101a4bd19062. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.759775 26792 tablet_service.cc:759] Processing CreateTablet for tablet 2b4d721d47104c6a9f7e101a4bd19062 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.759896 27210 ts_tablet_manager.cc:1382] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.760129 27210 log.cc:986] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/76754c71d69d462a83d462412180fb60
I0619 00:12:17.760349 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b4d721d47104c6a9f7e101a4bd19062. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.760596 27210 ts_tablet_manager.cc:1403] T 76754c71d69d462a83d462412180fb60 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:17.761380 27196 tablet_bootstrap.cc:578] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:17.762348 27210 tablet_replica.cc:261] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:17.763165 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 76754c71d69d462a83d462412180fb60 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:17.769196 27063 tablet_service.cc:759] Processing CreateTablet for tablet 2b4d721d47104c6a9f7e101a4bd19062 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.769848 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b4d721d47104c6a9f7e101a4bd19062. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.771981 27062 ts_tablet_manager.cc:1173] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.772388 27195 ts_tablet_manager.cc:1028] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.772742 27195 tablet_bootstrap.cc:427] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:17.774215 27196 tablet_bootstrap.cc:427] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:17.774531 27196 ts_tablet_manager.cc:1045] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.003s
I0619 00:12:17.774996 27195 tablet_bootstrap.cc:578] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:17.775916 27196 raft_consensus.cc:325] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.776530 27196 raft_consensus.cc:351] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.776743 27196 raft_consensus.cc:628] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:17.777190 27196 consensus_queue.cc:227] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.778828 27063 ts_tablet_manager.cc:1173] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.779422 27196 ts_tablet_manager.cc:1073] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
W0619 00:12:17.781023 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.783670 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.783951 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.780608 27197 tablet_bootstrap.cc:427] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.784330 27197 ts_tablet_manager.cc:1045] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.034s	user 0.005s	sys 0.004s
I0619 00:12:17.770517 27200 ts_tablet_manager.cc:1382] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.784904 27200 log.cc:986] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/76754c71d69d462a83d462412180fb60
I0619 00:12:17.785673 27200 ts_tablet_manager.cc:1403] T 76754c71d69d462a83d462412180fb60 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:17.786387 27197 raft_consensus.cc:325] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.787189 27197 raft_consensus.cc:351] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.787403 27197 raft_consensus.cc:628] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.787889 27197 consensus_queue.cc:227] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.788641 27176 thread.cc:554] rpc reactor (reactor) Time spent creating pthread: real 0.531s	user 0.014s	sys 0.099s
I0619 00:12:17.789613 27197 ts_tablet_manager.cc:1073] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
W0619 00:12:17.789742 27176 thread.cc:521] rpc reactor (reactor) Time spent starting thread: real 0.532s	user 0.016s	sys 0.099s
I0619 00:12:17.790076 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
W0619 00:12:17.790243 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.790514 27200 tablet_replica.cc:261] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:17.790969 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.791184 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.791386 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.791618 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.791853 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
W0619 00:12:17.791844 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.792073 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.792271 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.792171 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 76754c71d69d462a83d462412180fb60 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:17.792469 27200 raft_consensus.cc:2042] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.792764 27200 raft_consensus.cc:2069] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.793927 27200 ts_tablet_manager.cc:1369] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.795130 26792 ts_tablet_manager.cc:1173] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.795538 27196 ts_tablet_manager.cc:1028] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:17.795908 27196 tablet_bootstrap.cc:427] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:17.795919 27195 tablet_bootstrap.cc:427] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:17.796227 27195 ts_tablet_manager.cc:1045] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.024s	user 0.006s	sys 0.002s
I0619 00:12:17.797302 27199 ts_tablet_manager.cc:1382] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:17.797549 27195 raft_consensus.cc:325] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.798202 27195 raft_consensus.cc:351] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.798380 27195 raft_consensus.cc:628] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:17.798543 27196 tablet_bootstrap.cc:578] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:17.798995 27195 consensus_queue.cc:227] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.801703 27195 ts_tablet_manager.cc:1073] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I0619 00:12:17.790303 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.802016 27195 ts_tablet_manager.cc:1028] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.802151 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.802420 27195 tablet_bootstrap.cc:427] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:17.802819 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.804006 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.804245 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.804466 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.804697 27210 raft_consensus.cc:2042] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.804978 27210 raft_consensus.cc:2069] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:17.805758 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.806780 27210 ts_tablet_manager.cc:1369] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.797538 27199 log.cc:986] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/76754c71d69d462a83d462412180fb60
I0619 00:12:17.779839 26928 ts_tablet_manager.cc:1173] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:17.809538 27015 consensus_peers.cc:434] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Couldn't send request to peer 10c4e713d4c44dd8904e5f77ba815785 for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:17.810125 27197 ts_tablet_manager.cc:1028] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.810475 27197 tablet_bootstrap.cc:427] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.810449 27256 raft_consensus.cc:457] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.810760 27256 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.156s (starting election)
I0619 00:12:17.811017 27256 raft_consensus.cc:479] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.812090 27256 leader_election.cc:243] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.812501 27256 leader_election.cc:243] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.812703 27197 tablet_bootstrap.cc:578] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.814285 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7aff82640ec542be9aad6a4e782e45a8" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.814786 26949 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.977s (vote granted)
I0619 00:12:17.815026 26949 raft_consensus.cc:2245] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:17.816520 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7aff82640ec542be9aad6a4e782e45a8" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:17.817035 27083 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.907s (vote granted)
I0619 00:12:17.817142 26732 leader_election.cc:379] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.817265 27083 raft_consensus.cc:2245] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:17.817360 26732 leader_election.cc:270] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.818260 27256 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.006s (election complete)
I0619 00:12:17.818502 27256 raft_consensus.cc:2539] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.818729 27256 raft_consensus.cc:457] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.818940 27256 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.811s (starting election)
I0619 00:12:17.819151 27256 raft_consensus.cc:2784] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.820327 26732 leader_election.cc:379] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.825803 27195 tablet_bootstrap.cc:578] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:17.826253 27199 ts_tablet_manager.cc:1403] T 76754c71d69d462a83d462412180fb60 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:17.826318 27196 tablet_bootstrap.cc:427] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:17.826714 27196 ts_tablet_manager.cc:1045] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.031s	user 0.004s	sys 0.002s
I0619 00:12:17.828037 27196 raft_consensus.cc:325] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.828629 27196 raft_consensus.cc:351] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.828842 27196 raft_consensus.cc:628] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:17.829296 27196 consensus_queue.cc:227] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.836081 27013 consensus_peers.cc:434] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Couldn't send request to peer 1f719e8be9c9447a9acd40728c0a5710 for tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:17.836220 27256 raft_consensus.cc:479] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.837231 27256 leader_election.cc:243] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.838281 27256 leader_election.cc:243] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.838542 27197 tablet_bootstrap.cc:427] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:17.838907 27197 ts_tablet_manager.cc:1045] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.029s	user 0.005s	sys 0.001s
I0619 00:12:17.840593 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7aff82640ec542be9aad6a4e782e45a8" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:17.841063 26949 raft_consensus.cc:2784] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.841295 26949 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.974s (vote granted)
I0619 00:12:17.842170 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 76754c71d69d462a83d462412180fb60 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:17.842053 27197 raft_consensus.cc:325] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.842823 27197 raft_consensus.cc:351] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.843300 27197 raft_consensus.cc:628] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:17.844245 27197 consensus_queue.cc:227] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.845193 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.846467 27199 tablet_replica.cc:261] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:17.846820 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.847057 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.847266 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.847481 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.847738 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
W0619 00:12:17.847743 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.847975 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.848191 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(5a63eb3f8e3e48cd9d0ceb3a81e901b0)
I0619 00:12:17.848419 27199 raft_consensus.cc:2042] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:17.848667 27199 consensus_peers.cc:456] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
W0619 00:12:17.852535 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.855003 27200 ts_tablet_manager.cc:1382] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.854939 26949 raft_consensus.cc:2245] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:17.855273 27200 log.cc:986] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/5a63eb3f8e3e48cd9d0ceb3a81e901b0
I0619 00:12:17.855777 27200 ts_tablet_manager.cc:1403] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:17.856123 26732 leader_election.cc:379] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.856370 26732 leader_election.cc:270] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.856948 27199 consensus_peers.cc:456] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:17.846820 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7aff82640ec542be9aad6a4e782e45a8" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:17.857882 27083 raft_consensus.cc:2784] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.858139 27083 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.707s (vote granted)
I0619 00:12:17.859858 27256 raft_consensus.cc:2732] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.532s (election complete)
I0619 00:12:17.860095 27256 raft_consensus.cc:2539] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.860332 27199 pending_rounds.cc:63] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:17.860378 27256 raft_consensus.cc:593] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:17.860502 27199 pending_rounds.cc:68] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391013446795264 op_type: NO_OP noop_request { }
I0619 00:12:17.860800 27199 raft_consensus.cc:2622] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:17.860975 27197 ts_tablet_manager.cc:1073] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.001s
I0619 00:12:17.861011 27199 raft_consensus.cc:2069] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.860828 27256 consensus_queue.cc:204] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:17.862118 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:17.864058 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.864512 27199 ts_tablet_manager.cc:1369] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.865624 27083 raft_consensus.cc:2245] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:17.867528 27210 ts_tablet_manager.cc:1382] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:17.869288 27210 log.cc:986] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/5a63eb3f8e3e48cd9d0ceb3a81e901b0
I0619 00:12:17.869405 27195 tablet_bootstrap.cc:427] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:17.870007 27210 ts_tablet_manager.cc:1403] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
W0619 00:12:17.870016 26524 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000): Couldn't send request to peer b8582cfc277d4d4597c5605b7ac0ed48 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times.
I0619 00:12:17.870394 27195 ts_tablet_manager.cc:1045] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.068s	user 0.005s	sys 0.004s
I0619 00:12:17.871767 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:17.871697 27195 raft_consensus.cc:325] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.872320 27195 raft_consensus.cc:351] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:17.872515 27195 raft_consensus.cc:628] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:17.872961 27195 consensus_queue.cc:227] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.875427 27195 ts_tablet_manager.cc:1073] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
W0619 00:12:17.876811 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.885577 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:17.886402 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.886837 27259 raft_consensus.cc:457] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:17.887624 27259 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.102s (starting election)
I0619 00:12:17.887938 27259 raft_consensus.cc:479] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.892870 27259 leader_election.cc:243] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.896041 26732 leader_election.cc:379] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.896466 27196 ts_tablet_manager.cc:1073] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.070s	user 0.004s	sys 0.011s
I0619 00:12:17.897986 27083 raft_consensus.cc:1081] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:17.895253 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.901180 27259 leader_election.cc:243] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.901175 27287 consensus_queue.cc:963] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W0619 00:12:17.901749 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.904578 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b4d721d47104c6a9f7e101a4bd19062" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:17.905139 27083 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.902s (vote granted)
I0619 00:12:17.905357 27083 raft_consensus.cc:2245] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:17.906049 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b4d721d47104c6a9f7e101a4bd19062" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:17.906536 26949 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.873s (vote granted)
I0619 00:12:17.906797 26949 raft_consensus.cc:2245] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
W0619 00:12:17.908457 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.909116 26732 leader_election.cc:379] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.909445 26732 leader_election.cc:270] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:17.909910 27287 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.956s (election complete)
I0619 00:12:17.910164 27287 raft_consensus.cc:2539] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:17.910377 27287 raft_consensus.cc:457] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:17.910606 27287 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.077s (starting election)
I0619 00:12:17.910814 27287 raft_consensus.cc:2784] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.916347 26732 leader_election.cc:379] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.917644 27199 ts_tablet_manager.cc:1382] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:17.917906 27199 log.cc:986] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/5a63eb3f8e3e48cd9d0ceb3a81e901b0
I0619 00:12:17.918509 27199 ts_tablet_manager.cc:1403] T 5a63eb3f8e3e48cd9d0ceb3a81e901b0 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:17.920367 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 5a63eb3f8e3e48cd9d0ceb3a81e901b0 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:17.921352 27287 raft_consensus.cc:479] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.922375 27287 leader_election.cc:243] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.922930 27287 leader_election.cc:243] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.924600 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b4d721d47104c6a9f7e101a4bd19062" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:17.925410 26949 raft_consensus.cc:2784] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.925621 26949 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.850s (vote granted)
W0619 00:12:17.928820 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.933068 26949 raft_consensus.cc:2245] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:17.937793 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b4d721d47104c6a9f7e101a4bd19062" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:17.938242 27083 raft_consensus.cc:2784] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:17.938482 27083 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.058s (vote granted)
I0619 00:12:17.938616 26732 leader_election.cc:379] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:17.938999 26732 leader_election.cc:270] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:17.939764 27256 raft_consensus.cc:2732] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.052s (election complete)
I0619 00:12:17.940024 27256 raft_consensus.cc:2539] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:17.940347 27256 raft_consensus.cc:593] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:17.942297 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 7aff82640ec542be9aad6a4e782e45a8 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:17.940827 27256 consensus_queue.cc:204] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:17.946467 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet e5c538f18f88491787160e68800e29d7 from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:17.950314 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:17.955101 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 7aff82640ec542be9aad6a4e782e45a8
I0619 00:12:17.955391 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7aff82640ec542be9aad6a4e782e45a8 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:17.955806 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7aff82640ec542be9aad6a4e782e45a8 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:17.956094 27083 raft_consensus.cc:2245] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:17.956215 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7aff82640ec542be9aad6a4e782e45a8 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:17.956526 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 7aff82640ec542be9aad6a4e782e45a8 deleted and replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c
I0619 00:12:17.957535 26533 catalog_manager.cc:3663] Got report from deleted tablet 7aff82640ec542be9aad6a4e782e45a8 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c): Sending delete request for this tablet
I0619 00:12:17.958158 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7aff82640ec542be9aad6a4e782e45a8 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:17.959380 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 7aff82640ec542be9aad6a4e782e45a8 with delete_type TABLET_DATA_DELETED (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:17.959460 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 7aff82640ec542be9aad6a4e782e45a8 with delete_type TABLET_DATA_DELETED (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.960037 27210 tablet_replica.cc:261] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:17.960270 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.960472 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.960682 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.960618 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 7aff82640ec542be9aad6a4e782e45a8 with delete_type TABLET_DATA_DELETED (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.960877 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.960824 26790 tablet_service.cc:800] Processing DeleteTablet for tablet 7aff82640ec542be9aad6a4e782e45a8 with delete_type TABLET_DATA_DELETED (replaced by tablet f6f9d8a0a266441bb48b4be5a899ca5c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.961097 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.961073 27200 tablet_replica.cc:261] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:17.961290 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.961308 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.961470 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.961509 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.961642 27210 raft_consensus.cc:2042] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.961722 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.961915 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.961930 27210 raft_consensus.cc:2069] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.962128 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.962321 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.962507 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.962705 27200 raft_consensus.cc:2042] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:17.962906 27200 consensus_peers.cc:456] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:17.963150 27200 consensus_peers.cc:456] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:17.963469 27210 ts_tablet_manager.cc:1369] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.963503 27200 pending_rounds.cc:63] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:17.963675 27200 pending_rounds.cc:68] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391015884177408 op_type: NO_OP noop_request { }
I0619 00:12:17.963984 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet e5c538f18f88491787160e68800e29d7
I0619 00:12:17.963996 27200 raft_consensus.cc:2622] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:17.964217 27200 raft_consensus.cc:2069] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.964279 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5c538f18f88491787160e68800e29d7 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:17.964663 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5c538f18f88491787160e68800e29d7 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:17.965941 26790 tablet_service.cc:800] Processing DeleteTablet for tablet e5c538f18f88491787160e68800e29d7 with delete_type TABLET_DATA_DELETED (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:17.966599 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5c538f18f88491787160e68800e29d7 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:17.966962 26532 catalog_manager.cc:4596] ReplaceTablet: tablet e5c538f18f88491787160e68800e29d7 deleted and replaced by tablet fd5956f3fc424d32ae0e49aee743cf60
I0619 00:12:17.967182 26928 tablet_service.cc:800] Processing DeleteTablet for tablet e5c538f18f88491787160e68800e29d7 with delete_type TABLET_DATA_DELETED (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:17.967720 27063 tablet_service.cc:800] Processing DeleteTablet for tablet e5c538f18f88491787160e68800e29d7 with delete_type TABLET_DATA_DELETED (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:17.967779 26732 leader_election.cc:379] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:17.974148 27063 tablet_service.cc:759] Processing CreateTablet for tablet f6f9d8a0a266441bb48b4be5a899ca5c (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.974202 26928 tablet_service.cc:759] Processing CreateTablet for tablet f6f9d8a0a266441bb48b4be5a899ca5c (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.974769 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f6f9d8a0a266441bb48b4be5a899ca5c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.974800 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f6f9d8a0a266441bb48b4be5a899ca5c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.976368 26790 tablet_service.cc:759] Processing CreateTablet for tablet f6f9d8a0a266441bb48b4be5a899ca5c (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:17.979743 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f6f9d8a0a266441bb48b4be5a899ca5c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:17.984333 27200 ts_tablet_manager.cc:1369] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.987162 26928 ts_tablet_manager.cc:1173] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.987486 27199 tablet_replica.cc:261] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:17.987550 27197 ts_tablet_manager.cc:1028] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:17.987820 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.987917 27197 tablet_bootstrap.cc:427] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:17.988096 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.988315 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.988540 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.988819 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.989037 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.989913 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(7aff82640ec542be9aad6a4e782e45a8)
I0619 00:12:17.990108 27199 raft_consensus.cc:2042] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:17.990386 27199 raft_consensus.cc:2069] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:17.990855 26812 raft_consensus.cc:1081] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:17.992655 27206 consensus_queue.cc:963] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:17.993314 27199 ts_tablet_manager.cc:1369] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:17.996460 27197 tablet_bootstrap.cc:578] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:17.998361 27063 ts_tablet_manager.cc:1173] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:17.998762 27195 ts_tablet_manager.cc:1028] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:17.999107 27195 tablet_bootstrap.cc:427] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:17.999447 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.001338 27195 tablet_bootstrap.cc:578] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:18.003368 27210 ts_tablet_manager.cc:1382] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.002887 26532 catalog_manager.cc:3786] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:18.006027 27210 log.cc:986] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/7aff82640ec542be9aad6a4e782e45a8
I0619 00:12:18.006258 26792 tablet_service.cc:759] Processing CreateTablet for tablet fd5956f3fc424d32ae0e49aee743cf60 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.006897 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fd5956f3fc424d32ae0e49aee743cf60. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.007815 27210 ts_tablet_manager.cc:1403] T 7aff82640ec542be9aad6a4e782e45a8 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.009835 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 7aff82640ec542be9aad6a4e782e45a8 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:18.004142 26928 tablet_service.cc:759] Processing CreateTablet for tablet fd5956f3fc424d32ae0e49aee743cf60 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.010174 27197 tablet_bootstrap.cc:427] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.010496 27197 ts_tablet_manager.cc:1045] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.023s	user 0.003s	sys 0.005s
I0619 00:12:18.010720 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fd5956f3fc424d32ae0e49aee743cf60. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.011934 27197 raft_consensus.cc:325] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.013062 27063 tablet_service.cc:759] Processing CreateTablet for tablet fd5956f3fc424d32ae0e49aee743cf60 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.013690 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fd5956f3fc424d32ae0e49aee743cf60. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.012562 27197 raft_consensus.cc:351] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.016803 27197 raft_consensus.cc:628] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:18.017246 27197 consensus_queue.cc:227] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.020102 27197 ts_tablet_manager.cc:1073] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
W0619 00:12:18.019881 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.021226 27195 tablet_bootstrap.cc:427] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.021580 27195 ts_tablet_manager.cc:1045] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.005s
W0619 00:12:18.021950 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.023025 27195 raft_consensus.cc:325] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.023658 27195 raft_consensus.cc:351] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.023838 27195 raft_consensus.cc:628] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.024235 27195 consensus_queue.cc:227] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.026213 27210 tablet_replica.cc:261] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:18.026392 27195 ts_tablet_manager.cc:1073] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I0619 00:12:18.026486 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.026782 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.027016 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(e5c538f18f88491787160e68800e29d7)
W0619 00:12:18.027079 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.027240 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.027456 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.027737 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.027958 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.028105 27200 ts_tablet_manager.cc:1382] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.028165 27210 raft_consensus.cc:2042] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.028458 27210 raft_consensus.cc:2069] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:18.028731 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.029744 27210 ts_tablet_manager.cc:1369] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.037441 27200 log.cc:986] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/7aff82640ec542be9aad6a4e782e45a8
I0619 00:12:18.016659 26792 ts_tablet_manager.cc:1173] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.038170 27200 ts_tablet_manager.cc:1403] T 7aff82640ec542be9aad6a4e782e45a8 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.038563 26790 ts_tablet_manager.cc:1173] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.039350 27196 ts_tablet_manager.cc:1028] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.039773 27196 tablet_bootstrap.cc:427] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:18.042598 27063 ts_tablet_manager.cc:1173] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.042668 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 7aff82640ec542be9aad6a4e782e45a8 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
W0619 00:12:18.043819 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet 7aff82640ec542be9aad6a4e782e45a8 because the tablet was not found. No further retry: Not found: Tablet not found: 7aff82640ec542be9aad6a4e782e45a8
I0619 00:12:18.044667 27200 tablet_replica.cc:261] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.044749 27196 tablet_bootstrap.cc:578] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.050263 26532 catalog_manager.cc:3663] Got report from deleted tablet e5c538f18f88491787160e68800e29d7 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60): Sending delete request for this tablet
I0619 00:12:18.050962 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5c538f18f88491787160e68800e29d7 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60)
W0619 00:12:18.053462 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.055460 27195 ts_tablet_manager.cc:1028] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:18.055858 27195 tablet_bootstrap.cc:427] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:18.056462 27015 consensus_peers.cc:434] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Couldn't send request to peer 10c4e713d4c44dd8904e5f77ba815785 for tablet e5c538f18f88491787160e68800e29d7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:18.058202 27195 tablet_bootstrap.cc:578] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:18.059384 27252 raft_consensus.cc:457] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.059639 27252 raft_consensus.cc:2732] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.957s (starting election)
I0619 00:12:18.045102 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.059891 27252 raft_consensus.cc:479] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.060869 27252 leader_election.cc:243] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.061249 27252 leader_election.cc:243] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.061548 26532 catalog_manager.cc:3663] Got report from deleted tablet e5c538f18f88491787160e68800e29d7 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60): Sending delete request for this tablet
I0619 00:12:18.062366 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5c538f18f88491787160e68800e29d7 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.063369 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f6f9d8a0a266441bb48b4be5a899ca5c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.065102 27062 tablet_service.cc:800] Processing DeleteTablet for tablet e5c538f18f88491787160e68800e29d7 with delete_type TABLET_DATA_DELETED (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:18.065737 27015 leader_election.cc:306] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:18.066334 26790 tablet_service.cc:800] Processing DeleteTablet for tablet e5c538f18f88491787160e68800e29d7 with delete_type TABLET_DATA_DELETED (replaced by tablet fd5956f3fc424d32ae0e49aee743cf60) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.048297 26928 ts_tablet_manager.cc:1173] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.067003 27197 ts_tablet_manager.cc:1028] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.066913 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f6f9d8a0a266441bb48b4be5a899ca5c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:18.067390 27197 tablet_bootstrap.cc:427] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.067428 26950 raft_consensus.cc:2732] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.737s (vote granted)
I0619 00:12:18.068222 26950 raft_consensus.cc:2245] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:18.069046 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.069315 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.069804 27197 tablet_bootstrap.cc:578] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.070070 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.070324 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(e5c538f18f88491787160e68800e29d7)
W0619 00:12:18.071084 27013 consensus_peers.cc:434] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Couldn't send request to peer 1f719e8be9c9447a9acd40728c0a5710 for tablet e5c538f18f88491787160e68800e29d7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:18.071404 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.071645 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.071679 27013 leader_election.cc:379] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.071882 27200 raft_consensus.cc:2042] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.071897 27013 leader_election.cc:270] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.072166 27200 pending_rounds.cc:63] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:18.072635 27252 raft_consensus.cc:2732] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.021s (election complete)
I0619 00:12:18.072881 27252 raft_consensus.cc:2539] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.072872 27200 pending_rounds.cc:68] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391014812065792 op_type: NO_OP noop_request { }
I0619 00:12:18.073076 27252 raft_consensus.cc:457] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.073266 27252 raft_consensus.cc:2732] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.654s (starting election)
I0619 00:12:18.073443 27252 raft_consensus.cc:2784] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.073297 27200 raft_consensus.cc:2622] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:18.073822 27210 ts_tablet_manager.cc:1382] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.074053 27200 raft_consensus.cc:2069] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.077848 27200 ts_tablet_manager.cc:1369] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.077790 27252 raft_consensus.cc:479] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.078687 27252 leader_election.cc:243] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.079064 27252 leader_election.cc:243] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.079867 27210 log.cc:986] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/e5c538f18f88491787160e68800e29d7
I0619 00:12:18.080365 27210 ts_tablet_manager.cc:1403] T e5c538f18f88491787160e68800e29d7 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.081316 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f6f9d8a0a266441bb48b4be5a899ca5c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:18.081758 26950 raft_consensus.cc:2784] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.082001 26950 raft_consensus.cc:2732] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.196s (vote granted)
I0619 00:12:18.083894 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet e5c538f18f88491787160e68800e29d7 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:18.085846 27195 tablet_bootstrap.cc:427] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.086171 27195 ts_tablet_manager.cc:1045] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.030s	user 0.004s	sys 0.002s
I0619 00:12:18.086134 26950 raft_consensus.cc:2245] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:18.080811 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f6f9d8a0a266441bb48b4be5a899ca5c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.082484 27196 tablet_bootstrap.cc:427] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.088382 27196 ts_tablet_manager.cc:1045] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.049s	user 0.003s	sys 0.006s
I0619 00:12:18.089990 27196 raft_consensus.cc:325] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.095759 27196 raft_consensus.cc:351] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.095978 27196 raft_consensus.cc:628] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.096230 27199 ts_tablet_manager.cc:1382] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.096637 27199 log.cc:986] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/7aff82640ec542be9aad6a4e782e45a8
I0619 00:12:18.096508 27196 consensus_queue.cc:227] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.096693 27195 raft_consensus.cc:325] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.097311 27195 raft_consensus.cc:351] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.097368 27199 ts_tablet_manager.cc:1403] T 7aff82640ec542be9aad6a4e782e45a8 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.097509 27195 raft_consensus.cc:628] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.097954 27195 consensus_queue.cc:227] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.099158 27197 tablet_bootstrap.cc:427] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.099472 27197 ts_tablet_manager.cc:1045] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.032s	user 0.004s	sys 0.007s
I0619 00:12:18.099918 27013 leader_election.cc:379] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.100178 27013 leader_election.cc:270] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:18.100016 27199 tablet_replica.cc:261] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:18.103871 27197 raft_consensus.cc:325] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:18.106487 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.106721 27197 raft_consensus.cc:351] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.106935 27197 raft_consensus.cc:628] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
W0619 00:12:18.111760 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.113325 27196 ts_tablet_manager.cc:1073] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.025s	user 0.002s	sys 0.001s
I0619 00:12:18.113716 27196 ts_tablet_manager.cc:1028] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.114074 27196 tablet_bootstrap.cc:427] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:18.116647 27196 tablet_bootstrap.cc:578] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.116598 27197 consensus_queue.cc:227] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.107362 27204 raft_consensus.cc:2732] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.790s (election complete)
W0619 00:12:18.119168 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.119277 27204 raft_consensus.cc:2539] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:18.119601 27204 raft_consensus.cc:593] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:18.120410 27222 raft_consensus.cc:457] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.120646 27222 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.720s (starting election)
I0619 00:12:18.120080 27204 consensus_queue.cc:204] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.120934 27222 raft_consensus.cc:479] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.121888 27222 leader_election.cc:243] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:18.109941 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.122279 27222 leader_election.cc:243] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.119011 27195 ts_tablet_manager.cc:1073] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.032s	user 0.004s	sys 0.000s
W0619 00:12:18.123914 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.100419 27015 leader_election.cc:306] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:18.126137 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 7aff82640ec542be9aad6a4e782e45a8 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:18.126533 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fd5956f3fc424d32ae0e49aee743cf60" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:18.126978 27083 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.091s (vote granted)
I0619 00:12:18.127194 27083 raft_consensus.cc:2245] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.118146 27197 ts_tablet_manager.cc:1073] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.000s
I0619 00:12:18.129393 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fd5956f3fc424d32ae0e49aee743cf60" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.129904 26812 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.548s (vote granted)
I0619 00:12:18.130151 26812 raft_consensus.cc:2245] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.131237 26876 leader_election.cc:379] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.131461 26876 leader_election.cc:270] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.131932 27222 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.237s (election complete)
I0619 00:12:18.132158 27222 raft_consensus.cc:2539] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.131814 26874 leader_election.cc:379] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.132344 27222 raft_consensus.cc:457] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.132536 27222 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.241s (starting election)
I0619 00:12:18.132751 27222 raft_consensus.cc:2784] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.132304 26533 catalog_manager.cc:3786] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:18.139045 27222 raft_consensus.cc:479] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.140058 27222 leader_election.cc:243] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.141358 27222 leader_election.cc:243] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.144107 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fd5956f3fc424d32ae0e49aee743cf60" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.144527 26812 raft_consensus.cc:2784] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.144795 26812 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.931s (vote granted)
I0619 00:12:18.116422 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.151221 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fd5956f3fc424d32ae0e49aee743cf60" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:18.151625 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.151648 27083 raft_consensus.cc:2784] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.151990 27083 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.817s (vote granted)
I0619 00:12:18.153496 26812 raft_consensus.cc:2245] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.154695 26876 leader_election.cc:379] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.154924 26876 leader_election.cc:270] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:18.155421 27222 raft_consensus.cc:2732] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.501s (election complete)
I0619 00:12:18.155683 27222 raft_consensus.cc:2539] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:18.151928 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.156303 27196 tablet_bootstrap.cc:427] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.156399 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.156625 27196 ts_tablet_manager.cc:1045] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.043s	user 0.003s	sys 0.001s
I0619 00:12:18.156725 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.157002 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.157276 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(e5c538f18f88491787160e68800e29d7)
I0619 00:12:18.157496 27199 raft_consensus.cc:2042] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:18.157757 27199 consensus_peers.cc:456] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:18.157917 27196 raft_consensus.cc:325] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.158504 27196 raft_consensus.cc:351] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.158705 27196 raft_consensus.cc:628] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.158852 27199 consensus_peers.cc:456] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:18.159093 27196 consensus_queue.cc:227] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.160084 27199 raft_consensus.cc:2069] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:18.162705 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.164474 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.165310 27196 ts_tablet_manager.cc:1073] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.001s
I0619 00:12:18.166213 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet acdbab1409d24027954bfbf5bc07ee3c from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:18.166800 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.173403 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet fd5956f3fc424d32ae0e49aee743cf60 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:18.177855 27222 raft_consensus.cc:593] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:18.179750 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet acdbab1409d24027954bfbf5bc07ee3c
I0619 00:12:18.178401 27222 consensus_queue.cc:204] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.180049 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acdbab1409d24027954bfbf5bc07ee3c on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.180454 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acdbab1409d24027954bfbf5bc07ee3c on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.180850 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet acdbab1409d24027954bfbf5bc07ee3c on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.181174 26533 catalog_manager.cc:4596] ReplaceTablet: tablet acdbab1409d24027954bfbf5bc07ee3c deleted and replaced by tablet 5ae85377ea7f4ca8b17021ac0da064e4
I0619 00:12:18.182477 26790 tablet_service.cc:800] Processing DeleteTablet for tablet acdbab1409d24027954bfbf5bc07ee3c with delete_type TABLET_DATA_DELETED (replaced by tablet 5ae85377ea7f4ca8b17021ac0da064e4) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.189934 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet fd5956f3fc424d32ae0e49aee743cf60
I0619 00:12:18.190629 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fd5956f3fc424d32ae0e49aee743cf60 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.191602 26929 tablet_service.cc:800] Processing DeleteTablet for tablet fd5956f3fc424d32ae0e49aee743cf60 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.192096 27210 tablet_replica.cc:261] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:18.192327 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.192517 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.192708 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.192883 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.193089 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.193267 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.193470 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.193883 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fd5956f3fc424d32ae0e49aee743cf60 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.194265 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fd5956f3fc424d32ae0e49aee743cf60 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.194921 26532 catalog_manager.cc:4596] ReplaceTablet: tablet fd5956f3fc424d32ae0e49aee743cf60 deleted and replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:18.160167 27083 raft_consensus.cc:2245] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.191301 27062 tablet_service.cc:800] Processing DeleteTablet for tablet acdbab1409d24027954bfbf5bc07ee3c with delete_type TABLET_DATA_DELETED (replaced by tablet 5ae85377ea7f4ca8b17021ac0da064e4) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:18.183400 26928 tablet_service.cc:800] Processing DeleteTablet for tablet acdbab1409d24027954bfbf5bc07ee3c with delete_type TABLET_DATA_DELETED (replaced by tablet 5ae85377ea7f4ca8b17021ac0da064e4) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.195293 27063 tablet_service.cc:800] Processing DeleteTablet for tablet fd5956f3fc424d32ae0e49aee743cf60 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:18.196736 26874 leader_election.cc:379] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.198017 26790 tablet_service.cc:800] Processing DeleteTablet for tablet fd5956f3fc424d32ae0e49aee743cf60 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.199331 27199 ts_tablet_manager.cc:1369] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.205027 26928 tablet_service.cc:759] Processing CreateTablet for tablet 5ae85377ea7f4ca8b17021ac0da064e4 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.205623 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ae85377ea7f4ca8b17021ac0da064e4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.206866 26790 tablet_service.cc:759] Processing CreateTablet for tablet 5ae85377ea7f4ca8b17021ac0da064e4 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.207448 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ae85377ea7f4ca8b17021ac0da064e4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.209875 27210 raft_consensus.cc:2042] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:18.210049 27063 tablet_service.cc:759] Processing CreateTablet for tablet 5ae85377ea7f4ca8b17021ac0da064e4 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.210347 27210 consensus_peers.cc:456] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:18.210554 27210 consensus_peers.cc:456] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:18.210671 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ae85377ea7f4ca8b17021ac0da064e4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.210924 27210 pending_rounds.cc:63] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:18.211069 27210 pending_rounds.cc:68] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391017233784832 op_type: NO_OP noop_request { }
I0619 00:12:18.211360 27210 raft_consensus.cc:2622] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:18.211576 27210 raft_consensus.cc:2069] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.211832 26532 catalog_manager.cc:3663] Got report from deleted tablet fd5956f3fc424d32ae0e49aee743cf60 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1): Sending delete request for this tablet
I0619 00:12:18.212468 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fd5956f3fc424d32ae0e49aee743cf60 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.213393 26929 tablet_service.cc:800] Processing DeleteTablet for tablet fd5956f3fc424d32ae0e49aee743cf60 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f90ed424e734a78bc242e7d4708f1f1) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.215466 27200 ts_tablet_manager.cc:1382] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.215770 27200 log.cc:986] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/e5c538f18f88491787160e68800e29d7
I0619 00:12:18.216395 27200 ts_tablet_manager.cc:1403] T e5c538f18f88491787160e68800e29d7 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.220691 27062 tablet_service.cc:759] Processing CreateTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.221031 26929 tablet_service.cc:759] Processing CreateTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.221261 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1f90ed424e734a78bc242e7d4708f1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.221580 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1f90ed424e734a78bc242e7d4708f1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.223980 27210 ts_tablet_manager.cc:1369] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.224817 26792 tablet_service.cc:759] Processing CreateTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.225400 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1f90ed424e734a78bc242e7d4708f1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.232524 27062 ts_tablet_manager.cc:1173] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.234076 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet e5c538f18f88491787160e68800e29d7 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:18.234530 26792 ts_tablet_manager.cc:1173] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.234910 27196 ts_tablet_manager.cc:1028] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.235242 27196 tablet_bootstrap.cc:427] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:18.235643 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet e5c538f18f88491787160e68800e29d7 because the tablet was not found. No further retry: Not found: Tablet not found: e5c538f18f88491787160e68800e29d7
I0619 00:12:18.237476 27196 tablet_bootstrap.cc:578] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.238005 27195 ts_tablet_manager.cc:1028] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:18.238771 27195 tablet_bootstrap.cc:427] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:18.240397 27210 ts_tablet_manager.cc:1382] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.240613 27210 log.cc:986] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/fd5956f3fc424d32ae0e49aee743cf60
I0619 00:12:18.240783 27195 tablet_bootstrap.cc:578] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:18.241201 27210 ts_tablet_manager.cc:1403] T fd5956f3fc424d32ae0e49aee743cf60 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.242709 27063 ts_tablet_manager.cc:1173] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.243001 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet fd5956f3fc424d32ae0e49aee743cf60 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:18.244603 26790 ts_tablet_manager.cc:1173] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.244285 27200 tablet_replica.cc:261] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.246260 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.246706 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.247134 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.247607 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.248101 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.249815 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.250269 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.250679 27200 raft_consensus.cc:2042] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:18.250787 27196 tablet_bootstrap.cc:427] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.251154 27200 consensus_peers.cc:456] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:18.251307 27196 ts_tablet_manager.cc:1045] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.006s
I0619 00:12:18.251358 27200 consensus_peers.cc:456] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:18.252256 27200 raft_consensus.cc:2069] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.252609 27196 raft_consensus.cc:325] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.253219 27196 raft_consensus.cc:351] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.253388 27196 raft_consensus.cc:628] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.253692 26929 ts_tablet_manager.cc:1173] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.254050 27197 ts_tablet_manager.cc:1028] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.254379 27197 tablet_bootstrap.cc:427] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.256891 27196 consensus_queue.cc:227] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.257799 27197 tablet_bootstrap.cc:578] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.259876 27195 tablet_bootstrap.cc:427] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.260184 27195 ts_tablet_manager.cc:1045] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.003s
I0619 00:12:18.261907 27199 ts_tablet_manager.cc:1382] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.262161 27199 log.cc:986] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/e5c538f18f88491787160e68800e29d7
I0619 00:12:18.262754 27199 ts_tablet_manager.cc:1403] T e5c538f18f88491787160e68800e29d7 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.263006 27197 tablet_bootstrap.cc:427] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.263377 27197 ts_tablet_manager.cc:1045] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I0619 00:12:18.263535 27176 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:18.264613 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet e5c538f18f88491787160e68800e29d7 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:18.264789 27197 raft_consensus.cc:325] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.265461 27197 raft_consensus.cc:351] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.265679 27197 raft_consensus.cc:628] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:18.265631 27195 raft_consensus.cc:325] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.266227 27195 raft_consensus.cc:351] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.266404 27195 raft_consensus.cc:628] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.266268 27197 consensus_queue.cc:227] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.266903 27195 consensus_queue.cc:227] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.267396 27210 tablet_replica.cc:261] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
W0619 00:12:18.269995 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.270335 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.270567 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.270810 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(acdbab1409d24027954bfbf5bc07ee3c)
W0619 00:12:18.273367 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.275948 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.276254 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.276510 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.276764 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.276984 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.277199 27210 raft_consensus.cc:2042] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.274210 27197 ts_tablet_manager.cc:1073] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.000s
I0619 00:12:18.277501 27210 raft_consensus.cc:2069] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.280778 27195 ts_tablet_manager.cc:1073] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I0619 00:12:18.281831 27196 ts_tablet_manager.cc:1073] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.030s	user 0.004s	sys 0.000s
I0619 00:12:18.282179 27196 ts_tablet_manager.cc:1028] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.283077 26928 ts_tablet_manager.cc:1173] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:18.283336 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.285543 27197 ts_tablet_manager.cc:1028] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.285946 27197 tablet_bootstrap.cc:427] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.282546 27196 tablet_bootstrap.cc:427] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:18.289103 27176 master_main.cc:83] Starting Master server...
I0619 00:12:18.289793 27196 tablet_bootstrap.cc:578] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.289958 27197 tablet_bootstrap.cc:578] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.291446 27210 ts_tablet_manager.cc:1369] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.291666 27200 ts_tablet_manager.cc:1369] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:18.285336 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.294884 27195 ts_tablet_manager.cc:1028] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:18.295251 27195 tablet_bootstrap.cc:427] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:18.297688 27195 tablet_bootstrap.cc:578] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
W0619 00:12:18.298732 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet e5c538f18f88491787160e68800e29d7 because the tablet was not found. No further retry: Not found: Tablet not found: e5c538f18f88491787160e68800e29d7
I0619 00:12:18.300473 27196 tablet_bootstrap.cc:427] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.300818 27196 ts_tablet_manager.cc:1045] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.000s
I0619 00:12:18.302337 27196 raft_consensus.cc:325] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.302954 27196 raft_consensus.cc:351] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.303129 27196 raft_consensus.cc:628] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.303565 27196 consensus_queue.cc:227] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.305115 27196 ts_tablet_manager.cc:1073] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
W0619 00:12:18.307072 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.308743 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.298410 27199 tablet_replica.cc:261] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:18.309800 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.310050 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.310447 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.310338 27197 tablet_bootstrap.cc:427] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.310796 27197 ts_tablet_manager.cc:1045] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.001s
I0619 00:12:18.312358 27197 raft_consensus.cc:325] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.313000 27197 raft_consensus.cc:351] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.313201 27197 raft_consensus.cc:628] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:18.313668 27197 consensus_queue.cc:227] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:18.317126 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.317983 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.318248 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.318478 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.318714 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(acdbab1409d24027954bfbf5bc07ee3c)
I0619 00:12:18.318917 27199 raft_consensus.cc:2042] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
W0619 00:12:18.318964 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.319247 27199 raft_consensus.cc:2069] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.320423 27197 ts_tablet_manager.cc:1073] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I0619 00:12:18.304383 27210 ts_tablet_manager.cc:1382] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.321275 27210 log.cc:986] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/acdbab1409d24027954bfbf5bc07ee3c
I0619 00:12:18.321362 27199 ts_tablet_manager.cc:1369] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.323207 27210 ts_tablet_manager.cc:1403] T acdbab1409d24027954bfbf5bc07ee3c P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.325129 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet acdbab1409d24027954bfbf5bc07ee3c (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
W0619 00:12:18.325896 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet fd5956f3fc424d32ae0e49aee743cf60 because the tablet was not found. No further retry: Not found: Tablet not found: fd5956f3fc424d32ae0e49aee743cf60
I0619 00:12:18.332020 27195 tablet_bootstrap.cc:427] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.334700 27200 ts_tablet_manager.cc:1382] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.334960 27200 log.cc:986] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/acdbab1409d24027954bfbf5bc07ee3c
I0619 00:12:18.335139 27195 ts_tablet_manager.cc:1045] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.040s	user 0.004s	sys 0.002s
I0619 00:12:18.335510 27200 ts_tablet_manager.cc:1403] T acdbab1409d24027954bfbf5bc07ee3c P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.336503 27195 raft_consensus.cc:325] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.337129 27195 raft_consensus.cc:351] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.337388 27195 raft_consensus.cc:628] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.339287 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet acdbab1409d24027954bfbf5bc07ee3c (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:18.338153 27195 consensus_queue.cc:227] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.339733 27200 tablet_replica.cc:261] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.339969 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.340173 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(fd5956f3fc424d32ae0e49aee743cf60)
W0619 00:12:18.342212 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.344295 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.345674 27195 ts_tablet_manager.cc:1073] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I0619 00:12:18.340363 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.349892 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.350138 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.350338 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.350530 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.350736 27200 raft_consensus.cc:2042] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.351011 27200 raft_consensus.cc:2069] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.354161 27199 ts_tablet_manager.cc:1382] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.354408 27199 log.cc:986] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/acdbab1409d24027954bfbf5bc07ee3c
I0619 00:12:18.355010 27199 ts_tablet_manager.cc:1403] T acdbab1409d24027954bfbf5bc07ee3c P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.356895 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet acdbab1409d24027954bfbf5bc07ee3c (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:18.359925 27200 ts_tablet_manager.cc:1369] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.370287 27199 tablet_replica.cc:261] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
W0619 00:12:18.370643 26524 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000): Couldn't send request to peer b8582cfc277d4d4597c5605b7ac0ed48 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times.
I0619 00:12:18.371842 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.372059 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.372244 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.372433 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.372673 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.372879 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.373065 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(fd5956f3fc424d32ae0e49aee743cf60)
I0619 00:12:18.373245 27199 raft_consensus.cc:2042] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.373524 27199 raft_consensus.cc:2069] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.377060 27204 raft_consensus.cc:457] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.377279 27204 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.194s (starting election)
I0619 00:12:18.377512 27204 raft_consensus.cc:479] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.378466 27204 leader_election.cc:243] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.378870 27204 leader_election.cc:243] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.378898 27200 ts_tablet_manager.cc:1382] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.379115 27200 log.cc:986] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/fd5956f3fc424d32ae0e49aee743cf60
I0619 00:12:18.379345 27222 raft_consensus.cc:457] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.379585 27222 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.688s (starting election)
I0619 00:12:18.379871 27222 raft_consensus.cc:479] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.380620 27199 ts_tablet_manager.cc:1369] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.380852 27222 leader_election.cc:243] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.381232 27222 leader_election.cc:243] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.381129 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ae85377ea7f4ca8b17021ac0da064e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:18.381628 26950 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.575s (vote granted)
I0619 00:12:18.381870 26950 raft_consensus.cc:2245] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:18.382464 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ae85377ea7f4ca8b17021ac0da064e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.382946 26812 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.060s (vote granted)
I0619 00:12:18.383123 26812 raft_consensus.cc:2245] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:18.384083 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f90ed424e734a78bc242e7d4708f1f1" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:18.384531 27083 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.530s (vote granted)
I0619 00:12:18.385187 27083 raft_consensus.cc:2245] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.385848 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f90ed424e734a78bc242e7d4708f1f1" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.386283 26812 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.608s (vote granted)
I0619 00:12:18.386524 26812 raft_consensus.cc:2245] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.386947 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 2b4d721d47104c6a9f7e101a4bd19062 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:18.389199 26876 leader_election.cc:379] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.389422 26876 leader_election.cc:270] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.390766 27222 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.212s (election complete)
I0619 00:12:18.390990 27222 raft_consensus.cc:2539] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.391186 27222 raft_consensus.cc:457] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.391378 27222 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.928s (starting election)
I0619 00:12:18.391571 27222 raft_consensus.cc:2784] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.384244 27015 leader_election.cc:379] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.395464 27015 leader_election.cc:270] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.396749 26874 leader_election.cc:379] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.397366 27013 leader_election.cc:379] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.397492 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 2b4d721d47104c6a9f7e101a4bd19062
I0619 00:12:18.397806 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b4d721d47104c6a9f7e101a4bd19062 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.398211 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b4d721d47104c6a9f7e101a4bd19062 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.398612 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b4d721d47104c6a9f7e101a4bd19062 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.399581 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 2b4d721d47104c6a9f7e101a4bd19062 with delete_type TABLET_DATA_DELETED (replaced by tablet f56d681b86004d82a9a05094132c217c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.399953 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 2b4d721d47104c6a9f7e101a4bd19062 with delete_type TABLET_DATA_DELETED (replaced by tablet f56d681b86004d82a9a05094132c217c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.399993 27204 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.583s (election complete)
I0619 00:12:18.400239 27204 raft_consensus.cc:2539] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.400444 27204 raft_consensus.cc:457] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.400660 27204 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.504s (starting election)
I0619 00:12:18.400853 27204 raft_consensus.cc:2784] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.401748 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 2b4d721d47104c6a9f7e101a4bd19062 deleted and replaced by tablet f56d681b86004d82a9a05094132c217c
I0619 00:12:18.403606 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 2b4d721d47104c6a9f7e101a4bd19062 with delete_type TABLET_DATA_DELETED (replaced by tablet f56d681b86004d82a9a05094132c217c) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:18.403901 27210 tablet_replica.cc:261] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:18.404136 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.404331 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.404515 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.404633 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet fd5956f3fc424d32ae0e49aee743cf60 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:18.404731 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.404945 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.405133 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.405308 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.405483 27210 raft_consensus.cc:2042] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.405761 27210 raft_consensus.cc:2069] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.406973 27210 ts_tablet_manager.cc:1369] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.407609 27200 ts_tablet_manager.cc:1403] T fd5956f3fc424d32ae0e49aee743cf60 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.409581 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet fd5956f3fc424d32ae0e49aee743cf60 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:18.409811 27200 tablet_replica.cc:261] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.410045 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.410334 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.410537 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.410857 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.411180 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.411430 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.411664 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.411865 27200 raft_consensus.cc:2042] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:18.412075 27200 consensus_peers.cc:456] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:18.412276 27200 consensus_peers.cc:456] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:18.412617 27200 pending_rounds.cc:63] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:18.412760 27200 pending_rounds.cc:68] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391016264019968 op_type: NO_OP noop_request { }
I0619 00:12:18.413050 27200 raft_consensus.cc:2622] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:18.413269 27200 raft_consensus.cc:2069] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.420766 26792 tablet_service.cc:759] Processing CreateTablet for tablet f56d681b86004d82a9a05094132c217c (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.420930 26928 tablet_service.cc:759] Processing CreateTablet for tablet f56d681b86004d82a9a05094132c217c (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.421365 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f56d681b86004d82a9a05094132c217c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.421252 27062 tablet_service.cc:759] Processing CreateTablet for tablet f56d681b86004d82a9a05094132c217c (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.421512 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f56d681b86004d82a9a05094132c217c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.421836 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f56d681b86004d82a9a05094132c217c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.427098 27199 ts_tablet_manager.cc:1382] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.427377 27199 log.cc:986] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/fd5956f3fc424d32ae0e49aee743cf60
I0619 00:12:18.430045 27204 raft_consensus.cc:479] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.431236 27200 ts_tablet_manager.cc:1369] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.434674 26928 ts_tablet_manager.cc:1173] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.435056 27197 ts_tablet_manager.cc:1028] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.435380 27197 tablet_bootstrap.cc:427] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.437530 27197 tablet_bootstrap.cc:578] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.443042 27222 raft_consensus.cc:479] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.442739 27062 ts_tablet_manager.cc:1173] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.444069 27195 ts_tablet_manager.cc:1028] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:18.444262 27204 leader_election.cc:243] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.444411 27195 tablet_bootstrap.cc:427] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:18.445643 27222 leader_election.cc:243] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.446770 27195 tablet_bootstrap.cc:578] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:18.448051 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ae85377ea7f4ca8b17021ac0da064e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:18.448472 26950 raft_consensus.cc:2784] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.448734 26950 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.172s (vote granted)
I0619 00:12:18.442816 27199 ts_tablet_manager.cc:1403] T fd5956f3fc424d32ae0e49aee743cf60 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.451257 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet fd5956f3fc424d32ae0e49aee743cf60 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:18.453191 26950 raft_consensus.cc:2245] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:18.453378 27199 tablet_replica.cc:261] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:18.454236 27013 leader_election.cc:379] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.454474 27013 leader_election.cc:270] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:18.442548 26792 ts_tablet_manager.cc:1173] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.454931 27252 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.767s (election complete)
I0619 00:12:18.454875 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f90ed424e734a78bc242e7d4708f1f1" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.455098 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.455163 27252 raft_consensus.cc:2539] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:18.455301 26812 raft_consensus.cc:2784] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.455337 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.455250 27204 leader_election.cc:243] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.455508 27252 raft_consensus.cc:593] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:18.455534 26812 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.228s (vote granted)
I0619 00:12:18.456182 27252 consensus_queue.cc:204] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.455534 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(2b4d721d47104c6a9f7e101a4bd19062)
W0619 00:12:18.457600 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.457178 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.458072 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.458461 27222 leader_election.cc:243] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.459215 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.459247 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ae85377ea7f4ca8b17021ac0da064e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.459712 26811 raft_consensus.cc:2784] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.459954 26811 raft_consensus.cc:2732] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.004s (vote granted)
I0619 00:12:18.460319 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(2b4d721d47104c6a9f7e101a4bd19062)
I0619 00:12:18.460503 27199 raft_consensus.cc:2042] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.461303 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f90ed424e734a78bc242e7d4708f1f1" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:18.461724 27083 raft_consensus.cc:2784] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.461935 27083 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.041s (vote granted)
I0619 00:12:18.461922 27199 raft_consensus.cc:2069] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.463637 27199 ts_tablet_manager.cc:1369] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.465601 26812 raft_consensus.cc:2245] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.466282 27083 raft_consensus.cc:2245] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.467358 26874 leader_election.cc:379] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.467638 26874 leader_election.cc:270] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:18.467571 26811 raft_consensus.cc:2245] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:18.468467 27196 ts_tablet_manager.cc:1028] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.468863 27196 tablet_bootstrap.cc:427] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:18.468816 26876 leader_election.cc:379] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.469259 27197 tablet_bootstrap.cc:427] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.469583 27197 ts_tablet_manager.cc:1045] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.034s	user 0.004s	sys 0.002s
I0619 00:12:18.469569 27015 leader_election.cc:379] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.470849 27195 tablet_bootstrap.cc:427] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.471029 27196 tablet_bootstrap.cc:578] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.471155 27195 ts_tablet_manager.cc:1045] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.027s	user 0.004s	sys 0.005s
I0619 00:12:18.471228 27197 raft_consensus.cc:325] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.472003 27197 raft_consensus.cc:351] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.472214 27197 raft_consensus.cc:628] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:18.472837 27197 consensus_queue.cc:227] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.472949 27195 raft_consensus.cc:325] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.473721 27300 raft_consensus.cc:2732] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.934s (election complete)
I0619 00:12:18.473975 27300 raft_consensus.cc:2539] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:18.473970 27195 raft_consensus.cc:351] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.474287 27195 raft_consensus.cc:628] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.474836 27195 consensus_queue.cc:227] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.476284 27195 ts_tablet_manager.cc:1073] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
W0619 00:12:18.477291 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.478868 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.479378 27300 raft_consensus.cc:593] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:18.479857 27300 consensus_queue.cc:204] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:18.484593 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.487006 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.489317 27200 ts_tablet_manager.cc:1382] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.491513 27210 ts_tablet_manager.cc:1382] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:12:18.493309 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.494101 27197 ts_tablet_manager.cc:1073] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.000s
I0619 00:12:18.495270 27210 log.cc:986] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/2b4d721d47104c6a9f7e101a4bd19062
I0619 00:12:18.495824 27210 ts_tablet_manager.cc:1403] T 2b4d721d47104c6a9f7e101a4bd19062 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.495613 26533 catalog_manager.cc:3786] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:18.497762 27200 log.cc:986] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/2b4d721d47104c6a9f7e101a4bd19062
I0619 00:12:18.498702 27200 ts_tablet_manager.cc:1403] T 2b4d721d47104c6a9f7e101a4bd19062 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.500475 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 2b4d721d47104c6a9f7e101a4bd19062 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:18.501194 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 2b4d721d47104c6a9f7e101a4bd19062 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:18.501682 27196 tablet_bootstrap.cc:427] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.502010 27196 ts_tablet_manager.cc:1045] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.033s	user 0.005s	sys 0.006s
I0619 00:12:18.503377 27196 raft_consensus.cc:325] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.504019 27196 raft_consensus.cc:351] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.504215 27196 raft_consensus.cc:628] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.504789 27196 consensus_queue.cc:227] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:18.508432 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.512142 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.515399 27196 ts_tablet_manager.cc:1073] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I0619 00:12:18.517813 26811 raft_consensus.cc:2784] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.517702 26533 catalog_manager.cc:3786] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:18.522164 27300 raft_consensus.cc:457] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.522409 27300 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.227s (starting election)
I0619 00:12:18.522689 27300 raft_consensus.cc:479] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.523701 26811 raft_consensus.cc:1081] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:18.523715 27300 leader_election.cc:243] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.524116 27300 leader_election.cc:243] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.525876 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f56d681b86004d82a9a05094132c217c" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:18.526356 27083 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.500s (vote granted)
I0619 00:12:18.526571 27083 raft_consensus.cc:2245] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.526623 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f56d681b86004d82a9a05094132c217c" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.527791 26874 leader_election.cc:379] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.528023 26874 leader_election.cc:270] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.528313 26811 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.853s (vote granted)
I0619 00:12:18.528527 26811 raft_consensus.cc:2245] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.528797 27300 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.591s (election complete)
I0619 00:12:18.529032 27300 raft_consensus.cc:2539] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.529234 27300 raft_consensus.cc:457] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.529044 27252 consensus_queue.cc:963] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:18.529436 27300 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.700s (starting election)
I0619 00:12:18.529667 27300 raft_consensus.cc:2784] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.530378 26876 leader_election.cc:379] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:18.534076 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.541602 27199 ts_tablet_manager.cc:1382] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.541883 27199 log.cc:986] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/2b4d721d47104c6a9f7e101a4bd19062
I0619 00:12:18.542387 27199 ts_tablet_manager.cc:1403] T 2b4d721d47104c6a9f7e101a4bd19062 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.544142 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 2b4d721d47104c6a9f7e101a4bd19062 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:18.548393 26950 raft_consensus.cc:1081] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:18.551353 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.551745 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.553854 27275 consensus_queue.cc:963] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I0619 00:12:18.543751 27300 raft_consensus.cc:479] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.555207 27300 leader_election.cc:243] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.556857 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f56d681b86004d82a9a05094132c217c" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.557308 26811 raft_consensus.cc:2784] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.557551 26811 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.678s (vote granted)
W0619 00:12:18.563525 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.563520 26811 raft_consensus.cc:2245] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.556116 27300 leader_election.cc:243] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.564570 26876 leader_election.cc:379] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.564831 26876 leader_election.cc:270] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:18.565299 27328 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.866s (election complete)
I0619 00:12:18.565572 27328 raft_consensus.cc:2539] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:18.565887 27328 raft_consensus.cc:593] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:18.566733 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f56d681b86004d82a9a05094132c217c" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:18.566340 27328 consensus_queue.cc:204] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.567155 27083 raft_consensus.cc:2784] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.567395 27083 raft_consensus.cc:2732] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.792s (vote granted)
W0619 00:12:18.570401 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.571936 27083 raft_consensus.cc:2245] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.572983 26874 leader_election.cc:379] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
W0619 00:12:18.576195 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.581646 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:18.585486 26533 catalog_manager.cc:3786] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:18.607035 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet f56d681b86004d82a9a05094132c217c from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:18.610101 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 5ae85377ea7f4ca8b17021ac0da064e4 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:18.620239 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet f56d681b86004d82a9a05094132c217c
I0619 00:12:18.620471 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f56d681b86004d82a9a05094132c217c on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:18.620807 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f56d681b86004d82a9a05094132c217c on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:18.621186 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f56d681b86004d82a9a05094132c217c on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:18.621489 26533 catalog_manager.cc:4596] ReplaceTablet: tablet f56d681b86004d82a9a05094132c217c deleted and replaced by tablet e90d2c91db4f4d6d89983cfffa243761
I0619 00:12:18.621434 26792 tablet_service.cc:800] Processing DeleteTablet for tablet f56d681b86004d82a9a05094132c217c with delete_type TABLET_DATA_DELETED (replaced by tablet e90d2c91db4f4d6d89983cfffa243761) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.622382 27062 tablet_service.cc:800] Processing DeleteTablet for tablet f56d681b86004d82a9a05094132c217c with delete_type TABLET_DATA_DELETED (replaced by tablet e90d2c91db4f4d6d89983cfffa243761) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:18.622382 26928 tablet_service.cc:800] Processing DeleteTablet for tablet f56d681b86004d82a9a05094132c217c with delete_type TABLET_DATA_DELETED (replaced by tablet e90d2c91db4f4d6d89983cfffa243761) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.622894 27199 tablet_replica.cc:261] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:18.622889 27210 tablet_replica.cc:261] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:18.623127 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623127 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623325 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623325 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623404 27200 tablet_replica.cc:261] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.623514 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623514 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623654 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623725 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623725 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623832 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623939 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.623941 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624022 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624132 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624131 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624213 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624313 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624313 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624406 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624487 27199 raft_consensus.cc:2042] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.624485 27210 raft_consensus.cc:2042] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:18.624565 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624702 27210 consensus_peers.cc:456] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:18.624759 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(f56d681b86004d82a9a05094132c217c)
I0619 00:12:18.624778 27199 raft_consensus.cc:2069] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.624904 27210 consensus_peers.cc:456] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:18.624934 27200 raft_consensus.cc:2042] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.625180 27200 raft_consensus.cc:2069] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.625243 27210 pending_rounds.cc:63] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:18.625383 27210 pending_rounds.cc:68] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391018773884928 op_type: NO_OP noop_request { }
I0619 00:12:18.625715 27210 raft_consensus.cc:2622] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:18.625933 27210 raft_consensus.cc:2069] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.626489 27199 ts_tablet_manager.cc:1369] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.627243 27200 ts_tablet_manager.cc:1369] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.629525 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 5ae85377ea7f4ca8b17021ac0da064e4
I0619 00:12:18.629827 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5ae85377ea7f4ca8b17021ac0da064e4 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.630194 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5ae85377ea7f4ca8b17021ac0da064e4 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.630587 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5ae85377ea7f4ca8b17021ac0da064e4 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.630923 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 5ae85377ea7f4ca8b17021ac0da064e4 deleted and replaced by tablet 87b2d044789641668b76263c65f8eb1a
I0619 00:12:18.630920 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 5ae85377ea7f4ca8b17021ac0da064e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 87b2d044789641668b76263c65f8eb1a) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.632093 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 5ae85377ea7f4ca8b17021ac0da064e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 87b2d044789641668b76263c65f8eb1a) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:18.632939 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 5ae85377ea7f4ca8b17021ac0da064e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 87b2d044789641668b76263c65f8eb1a) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.642673 27200 ts_tablet_manager.cc:1382] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.643009 27210 ts_tablet_manager.cc:1369] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.648334 27199 ts_tablet_manager.cc:1382] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.651523 27200 log.cc:986] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/f56d681b86004d82a9a05094132c217c
I0619 00:12:18.652060 27200 ts_tablet_manager.cc:1403] T f56d681b86004d82a9a05094132c217c P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.653178 26792 tablet_service.cc:759] Processing CreateTablet for tablet e90d2c91db4f4d6d89983cfffa243761 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.653586 27199 log.cc:986] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/f56d681b86004d82a9a05094132c217c
I0619 00:12:18.649216 27062 tablet_service.cc:759] Processing CreateTablet for tablet e90d2c91db4f4d6d89983cfffa243761 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.653828 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e90d2c91db4f4d6d89983cfffa243761. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.654278 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e90d2c91db4f4d6d89983cfffa243761. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.647323 26928 tablet_service.cc:759] Processing CreateTablet for tablet e90d2c91db4f4d6d89983cfffa243761 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.656597 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e90d2c91db4f4d6d89983cfffa243761. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.663276 26792 ts_tablet_manager.cc:1173] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.663677 27196 ts_tablet_manager.cc:1028] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.664027 27196 tablet_bootstrap.cc:427] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:18.664865 27200 tablet_replica.cc:261] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.665102 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.665279 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.665442 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.665634 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.665829 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.665845 27199 ts_tablet_manager.cc:1403] T f56d681b86004d82a9a05094132c217c P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.665995 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.666164 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.666321 27200 raft_consensus.cc:2042] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.666558 27200 raft_consensus.cc:2069] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.667189 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet f56d681b86004d82a9a05094132c217c (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:18.667747 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet f56d681b86004d82a9a05094132c217c (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:18.669368 27196 tablet_bootstrap.cc:578] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.670958 26929 tablet_service.cc:759] Processing CreateTablet for tablet 87b2d044789641668b76263c65f8eb1a (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.671572 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 87b2d044789641668b76263c65f8eb1a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.671732 27063 tablet_service.cc:759] Processing CreateTablet for tablet 87b2d044789641668b76263c65f8eb1a (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.672291 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 87b2d044789641668b76263c65f8eb1a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.674247 27199 tablet_replica.cc:261] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:18.674491 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.674703 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.674924 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.675022 26790 tablet_service.cc:759] Processing CreateTablet for tablet 87b2d044789641668b76263c65f8eb1a (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.675149 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.675416 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.675603 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 87b2d044789641668b76263c65f8eb1a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.675633 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.675905 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.676110 27199 raft_consensus.cc:2042] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:18.676334 27199 consensus_peers.cc:456] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:18.676566 27199 consensus_peers.cc:456] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:18.676945 27199 pending_rounds.cc:63] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:18.677779 27199 pending_rounds.cc:68] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391018322239488 op_type: NO_OP noop_request { }
I0619 00:12:18.678118 27199 raft_consensus.cc:2622] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:18.679147 27199 raft_consensus.cc:2069] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.680364 26929 ts_tablet_manager.cc:1173] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.682255 27197 ts_tablet_manager.cc:1028] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.682603 27197 tablet_bootstrap.cc:427] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.685410 27197 tablet_bootstrap.cc:578] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.686952 27199 ts_tablet_manager.cc:1369] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.689873 27062 ts_tablet_manager.cc:1173] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.690286 27195 ts_tablet_manager.cc:1028] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:18.690618 27195 tablet_bootstrap.cc:427] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:18.692661 27200 ts_tablet_manager.cc:1369] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.693171 27195 tablet_bootstrap.cc:578] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:18.693457 27196 tablet_bootstrap.cc:427] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.693804 27196 ts_tablet_manager.cc:1045] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.030s	user 0.003s	sys 0.002s
I0619 00:12:18.695152 27196 raft_consensus.cc:325] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.695799 27196 raft_consensus.cc:351] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.696005 27196 raft_consensus.cc:628] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.696552 27196 consensus_queue.cc:227] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.698168 27063 ts_tablet_manager.cc:1173] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:18.699682 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.701745 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.702517 27196 ts_tablet_manager.cc:1073] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I0619 00:12:18.703855 27197 tablet_bootstrap.cc:427] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.704445 26790 ts_tablet_manager.cc:1173] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.704620 27197 ts_tablet_manager.cc:1045] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.005s
I0619 00:12:18.705994 27197 raft_consensus.cc:325] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.706712 27197 raft_consensus.cc:351] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.706918 27197 raft_consensus.cc:628] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:18.707456 27197 consensus_queue.cc:227] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:18.709931 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
W0619 00:12:18.711395 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.712070 27197 ts_tablet_manager.cc:1073] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I0619 00:12:18.716141 27196 ts_tablet_manager.cc:1028] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.716507 27196 tablet_bootstrap.cc:427] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:18.719270 27196 tablet_bootstrap.cc:578] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.720194 27195 tablet_bootstrap.cc:427] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.720563 27195 ts_tablet_manager.cc:1045] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.030s	user 0.003s	sys 0.006s
I0619 00:12:18.720669 26928 ts_tablet_manager.cc:1173] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.721050 27197 ts_tablet_manager.cc:1028] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.721401 27197 tablet_bootstrap.cc:427] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.722561 27195 raft_consensus.cc:325] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.723224 27195 raft_consensus.cc:351] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.723680 27210 ts_tablet_manager.cc:1382] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.723779 27197 tablet_bootstrap.cc:578] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.723956 27195 raft_consensus.cc:628] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.724076 27200 ts_tablet_manager.cc:1382] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.724316 27200 log.cc:986] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/5ae85377ea7f4ca8b17021ac0da064e4
I0619 00:12:18.727319 27195 consensus_queue.cc:227] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.728158 27200 ts_tablet_manager.cc:1403] T 5ae85377ea7f4ca8b17021ac0da064e4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
W0619 00:12:18.730279 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.730793 27195 ts_tablet_manager.cc:1073] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.001s
W0619 00:12:18.731981 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.733436 27196 tablet_bootstrap.cc:427] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.733775 27196 ts_tablet_manager.cc:1045] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.000s
I0619 00:12:18.735172 27196 raft_consensus.cc:325] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.735807 27196 raft_consensus.cc:351] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.735992 27196 raft_consensus.cc:628] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.736410 27196 consensus_queue.cc:227] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.737229 27210 log.cc:986] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/f56d681b86004d82a9a05094132c217c
I0619 00:12:18.737923 27210 ts_tablet_manager.cc:1403] T f56d681b86004d82a9a05094132c217c P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.737923 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 5ae85377ea7f4ca8b17021ac0da064e4 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
W0619 00:12:18.741145 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.731226 27195 ts_tablet_manager.cc:1028] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
W0619 00:12:18.743311 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.743932 27195 tablet_bootstrap.cc:427] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:18.745482 27197 tablet_bootstrap.cc:427] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.745700 27210 tablet_replica.cc:261] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:18.745857 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet f56d681b86004d82a9a05094132c217c (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:18.745945 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.745923 27197 ts_tablet_manager.cc:1045] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.009s
I0619 00:12:18.746217 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.746441 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.746661 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.746883 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.747023 27195 tablet_bootstrap.cc:578] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:18.747084 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.747269 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(5ae85377ea7f4ca8b17021ac0da064e4)
I0619 00:12:18.747445 27210 raft_consensus.cc:2042] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.747737 27210 raft_consensus.cc:2069] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.748394 27199 ts_tablet_manager.cc:1382] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.748654 27199 log.cc:986] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/5ae85377ea7f4ca8b17021ac0da064e4
I0619 00:12:18.749269 27199 ts_tablet_manager.cc:1403] T 5ae85377ea7f4ca8b17021ac0da064e4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.749599 27210 ts_tablet_manager.cc:1369] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.749233 27197 raft_consensus.cc:325] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.749954 27197 raft_consensus.cc:351] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.750155 27197 raft_consensus.cc:628] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:18.750587 27197 consensus_queue.cc:227] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.751185 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 5ae85377ea7f4ca8b17021ac0da064e4 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:18.752956 27196 ts_tablet_manager.cc:1073] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.000s
I0619 00:12:18.758174 27328 raft_consensus.cc:457] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.758430 27328 raft_consensus.cc:2732] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.790s (starting election)
W0619 00:12:18.758487 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.758714 27328 raft_consensus.cc:479] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.759781 27328 leader_election.cc:243] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.760203 27328 leader_election.cc:243] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.761628 27197 ts_tablet_manager.cc:1073] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.000s
I0619 00:12:18.762234 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87b2d044789641668b76263c65f8eb1a" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
W0619 00:12:18.762593 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.763957 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87b2d044789641668b76263c65f8eb1a" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.764439 26811 raft_consensus.cc:2732] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.514s (vote granted)
I0619 00:12:18.764695 26811 raft_consensus.cc:2245] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
W0619 00:12:18.765579 26874 leader_election.cc:306] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:18.766786 26876 leader_election.cc:379] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.767012 26876 leader_election.cc:270] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.767454 27328 raft_consensus.cc:2732] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.595s (election complete)
I0619 00:12:18.767714 27328 raft_consensus.cc:2539] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.767925 27328 raft_consensus.cc:457] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.768124 27328 raft_consensus.cc:2732] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.158s (starting election)
I0619 00:12:18.768330 27328 raft_consensus.cc:2784] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.756857 27195 tablet_bootstrap.cc:427] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.770112 27195 ts_tablet_manager.cc:1045] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.027s	user 0.005s	sys 0.000s
I0619 00:12:18.771490 27195 raft_consensus.cc:325] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.772311 27195 raft_consensus.cc:351] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.772521 27195 raft_consensus.cc:628] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.773360 27210 ts_tablet_manager.cc:1382] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.773015 27195 consensus_queue.cc:227] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.773715 27210 log.cc:986] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/5ae85377ea7f4ca8b17021ac0da064e4
W0619 00:12:18.775679 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.773429 27328 raft_consensus.cc:479] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.776290 27210 ts_tablet_manager.cc:1403] T 5ae85377ea7f4ca8b17021ac0da064e4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.776365 27328 leader_election.cc:243] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.776830 27328 leader_election.cc:243] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.778523 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87b2d044789641668b76263c65f8eb1a" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.778988 26811 raft_consensus.cc:2784] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.779238 26811 raft_consensus.cc:2732] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.647s (vote granted)
I0619 00:12:18.779258 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87b2d044789641668b76263c65f8eb1a" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:18.779717 27083 raft_consensus.cc:2784] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.779947 27083 raft_consensus.cc:2732] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.027s (vote granted)
I0619 00:12:18.783440 26811 raft_consensus.cc:2245] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.784443 27083 raft_consensus.cc:2245] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.784745 26876 leader_election.cc:379] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.784972 26876 leader_election.cc:270] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:18.785440 27328 raft_consensus.cc:2732] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.672s (election complete)
I0619 00:12:18.785676 27328 raft_consensus.cc:2539] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:18.785832 26874 leader_election.cc:379] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.785956 27328 raft_consensus.cc:593] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:18.786360 27328 consensus_queue.cc:204] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:18.788444 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.794183 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 5ae85377ea7f4ca8b17021ac0da064e4 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:18.796979 26533 catalog_manager.cc:3786] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
W0619 00:12:18.806797 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Network error: Failed to ping master at 127.24.119.61:43000: Client connection negotiation failed: client connection to 127.24.119.61:43000: connect: Connection refused (error 111)
I0619 00:12:18.811496 27195 ts_tablet_manager.cc:1073] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.041s	user 0.003s	sys 0.000s
I0619 00:12:18.814328 27176 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.61:43000
I0619 00:12:18.815661 27176 webserver.cc:175] Starting webserver on 127.24.119.61:40826
I0619 00:12:18.815827 27176 webserver.cc:186] Document root disabled
I0619 00:12:18.823109 27176 webserver.cc:313] Webserver started. Bound to: http://127.24.119.61:40826/
I0619 00:12:18.824123 27176 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-1/data/info.pb
I0619 00:12:18.826475 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 27176
I0619 00:12:18.832479 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 87b2d044789641668b76263c65f8eb1a from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:18.841336 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 1f90ed424e734a78bc242e7d4708f1f1 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:18.847455 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 87b2d044789641668b76263c65f8eb1a
I0619 00:12:18.847771 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 87b2d044789641668b76263c65f8eb1a on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:18.848161 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 87b2d044789641668b76263c65f8eb1a on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:18.848546 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 87b2d044789641668b76263c65f8eb1a on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:18.848896 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 87b2d044789641668b76263c65f8eb1a deleted and replaced by tablet b766f1ebdb404f5ab38c0ceedec69f37
I0619 00:12:18.848812 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 87b2d044789641668b76263c65f8eb1a with delete_type TABLET_DATA_DELETED (replaced by tablet b766f1ebdb404f5ab38c0ceedec69f37) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.849308 27210 tablet_replica.cc:261] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:18.849539 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.849759 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.849939 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.850124 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.850332 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.850523 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.850731 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.850908 27210 raft_consensus.cc:2042] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:18.851105 27210 consensus_peers.cc:456] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:18.851336 27210 consensus_peers.cc:456] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:18.851713 27210 pending_rounds.cc:63] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:18.851867 27210 pending_rounds.cc:68] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391019674447872 op_type: NO_OP noop_request { }
I0619 00:12:18.852174 27210 raft_consensus.cc:2622] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:18.854249 26790 tablet_service.cc:800] Processing DeleteTablet for tablet 87b2d044789641668b76263c65f8eb1a with delete_type TABLET_DATA_DELETED (replaced by tablet b766f1ebdb404f5ab38c0ceedec69f37) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.854751 27200 tablet_replica.cc:261] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.854987 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.855192 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.855362 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.855340 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 87b2d044789641668b76263c65f8eb1a with delete_type TABLET_DATA_DELETED (replaced by tablet b766f1ebdb404f5ab38c0ceedec69f37) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:18.855536 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.852401 27210 raft_consensus.cc:2069] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.855758 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.855948 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.856125 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.856292 27200 raft_consensus.cc:2042] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.856299 27199 tablet_replica.cc:261] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:18.856549 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.856549 27200 raft_consensus.cc:2069] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.856772 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.856958 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.857134 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.857352 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.857548 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.857758 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(87b2d044789641668b76263c65f8eb1a)
I0619 00:12:18.857954 27199 raft_consensus.cc:2042] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.858228 27199 raft_consensus.cc:2069] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.859025 27210 ts_tablet_manager.cc:1369] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.860493 27200 ts_tablet_manager.cc:1369] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.862221 27372 sys_catalog.cc:188] Verifying existing consensus state
I0619 00:12:18.862712 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:18.863009 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f90ed424e734a78bc242e7d4708f1f1 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:18.863389 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f90ed424e734a78bc242e7d4708f1f1 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:18.863813 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f90ed424e734a78bc242e7d4708f1f1 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:18.864156 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 1f90ed424e734a78bc242e7d4708f1f1 deleted and replaced by tablet 2b155ef1ed014ae5be07e75500df9659
I0619 00:12:18.864460 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b155ef1ed014ae5be07e75500df9659) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.865461 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b155ef1ed014ae5be07e75500df9659) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:18.866988 26790 tablet_service.cc:800] Processing DeleteTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b155ef1ed014ae5be07e75500df9659) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:18.869213 27083 raft_consensus.cc:1081] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:18.870537 27328 consensus_queue.cc:963] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:18.873682 27200 ts_tablet_manager.cc:1382] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.874030 27200 log.cc:986] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/87b2d044789641668b76263c65f8eb1a
I0619 00:12:18.875257 27200 ts_tablet_manager.cc:1403] T 87b2d044789641668b76263c65f8eb1a P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.880861 27200 tablet_replica.cc:261] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:18.881247 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.884121 26928 tablet_service.cc:759] Processing CreateTablet for tablet b766f1ebdb404f5ab38c0ceedec69f37 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.884749 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b766f1ebdb404f5ab38c0ceedec69f37. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.885236 27063 tablet_service.cc:759] Processing CreateTablet for tablet b766f1ebdb404f5ab38c0ceedec69f37 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.885869 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b766f1ebdb404f5ab38c0ceedec69f37. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.886257 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 87b2d044789641668b76263c65f8eb1a (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:18.887908 26532 catalog_manager.cc:3663] Got report from deleted tablet 1f90ed424e734a78bc242e7d4708f1f1 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659): Sending delete request for this tablet
I0619 00:12:18.888550 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f90ed424e734a78bc242e7d4708f1f1 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:18.890074 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:18.865499 27199 ts_tablet_manager.cc:1369] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.893718 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b155ef1ed014ae5be07e75500df9659) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:18.895671 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:18.882314 26790 tablet_service.cc:759] Processing CreateTablet for tablet b766f1ebdb404f5ab38c0ceedec69f37 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.897853 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b766f1ebdb404f5ab38c0ceedec69f37. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.906321 27063 ts_tablet_manager.cc:1173] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.906633 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.906745 27195 ts_tablet_manager.cc:1028] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:18.906950 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.907091 27195 tablet_bootstrap.cc:427] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:18.909356 27195 tablet_bootstrap.cc:578] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:18.909354 26792 tablet_service.cc:759] Processing CreateTablet for tablet 2b155ef1ed014ae5be07e75500df9659 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.909950 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b155ef1ed014ae5be07e75500df9659. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.910681 26929 tablet_service.cc:759] Processing CreateTablet for tablet 2b155ef1ed014ae5be07e75500df9659 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.911295 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b155ef1ed014ae5be07e75500df9659. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.915084 27199 ts_tablet_manager.cc:1382] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.917856 27328 raft_consensus.cc:457] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.918102 27328 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.586s (starting election)
I0619 00:12:18.918479 26792 ts_tablet_manager.cc:1173] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.918364 27328 raft_consensus.cc:479] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.918874 27196 ts_tablet_manager.cc:1028] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:18.919225 27196 tablet_bootstrap.cc:427] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:18.919405 27328 leader_election.cc:243] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.919864 27328 leader_election.cc:243] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.918599 27195 tablet_bootstrap.cc:427] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:18.920652 27195 ts_tablet_manager.cc:1045] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.001s
I0619 00:12:18.921901 27196 tablet_bootstrap.cc:578] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.921761 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e90d2c91db4f4d6d89983cfffa243761" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.922220 26811 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.917s (vote granted)
I0619 00:12:18.922442 26811 raft_consensus.cc:2245] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.923702 26876 leader_election.cc:379] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.923935 26876 leader_election.cc:270] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.923943 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e90d2c91db4f4d6d89983cfffa243761" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:18.924446 27082 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.701s (vote granted)
I0619 00:12:18.924711 27082 raft_consensus.cc:2245] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:18.925500 27195 raft_consensus.cc:325] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.925983 26874 leader_election.cc:379] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.926203 27195 raft_consensus.cc:351] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.926416 27195 raft_consensus.cc:628] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:18.926436 27328 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.170s (election complete)
I0619 00:12:18.926694 27328 raft_consensus.cc:2539] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.926882 27328 raft_consensus.cc:457] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.927062 27328 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.755s (starting election)
I0619 00:12:18.927242 27328 raft_consensus.cc:2784] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.926995 27195 consensus_queue.cc:227] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.928962 26928 ts_tablet_manager.cc:1173] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.929339 27197 ts_tablet_manager.cc:1028] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.929690 27197 tablet_bootstrap.cc:427] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.930636 27195 ts_tablet_manager.cc:1073] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I0619 00:12:18.931733 27199 log.cc:986] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/87b2d044789641668b76263c65f8eb1a
I0619 00:12:18.931995 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:18.932373 27210 ts_tablet_manager.cc:1382] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:18.932633 27210 log.cc:986] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/87b2d044789641668b76263c65f8eb1a
I0619 00:12:18.933194 27210 ts_tablet_manager.cc:1403] T 87b2d044789641668b76263c65f8eb1a P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:18.934092 27199 ts_tablet_manager.cc:1403] T 87b2d044789641668b76263c65f8eb1a P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:18.932341 27197 tablet_bootstrap.cc:578] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.937208 27328 raft_consensus.cc:479] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.938308 27328 leader_election.cc:243] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.938889 27328 leader_election.cc:243] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.940361 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e90d2c91db4f4d6d89983cfffa243761" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.940814 26811 raft_consensus.cc:2784] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.941056 26811 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.937s (vote granted)
I0619 00:12:18.942925 26929 ts_tablet_manager.cc:1173] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.907176 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.944077 27372 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Bootstrap starting.
I0619 00:12:18.944322 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.945220 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.945441 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.945639 27200 raft_consensus.cc:2042] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:18.945941 27200 raft_consensus.cc:2069] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:18.946205 26811 raft_consensus.cc:2245] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.947234 27200 ts_tablet_manager.cc:1369] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:18.947414 26876 leader_election.cc:379] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.947656 26876 leader_election.cc:270] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:18.950011 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 87b2d044789641668b76263c65f8eb1a (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:18.950922 26790 ts_tablet_manager.cc:1173] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:18.951874 27328 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.154s (election complete)
I0619 00:12:18.952160 27328 raft_consensus.cc:2539] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:18.952323 27210 tablet_replica.cc:261] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:18.957115 27328 raft_consensus.cc:593] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:18.957592 27328 consensus_queue.cc:204] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.958994 26533 catalog_manager.cc:3663] Got report from deleted tablet 1f90ed424e734a78bc242e7d4708f1f1 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659): Sending delete request for this tablet
I0619 00:12:18.959691 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f90ed424e734a78bc242e7d4708f1f1 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:18.961503 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b155ef1ed014ae5be07e75500df9659) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:18.954129 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.952028 27197 tablet_bootstrap.cc:427] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:18.965039 27197 ts_tablet_manager.cc:1045] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.035s	user 0.007s	sys 0.001s
W0619 00:12:18.964135 26876 consensus_peers.cc:434] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Couldn't send request to peer 10c4e713d4c44dd8904e5f77ba815785 for tablet 1f90ed424e734a78bc242e7d4708f1f1. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:18.965788 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.966027 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.966261 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.966506 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.966866 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.967085 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:18.967444 27210 raft_consensus.cc:2042] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
W0619 00:12:18.938048 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:18.940016 27063 tablet_service.cc:759] Processing CreateTablet for tablet 2b155ef1ed014ae5be07e75500df9659 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:18.944623 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e90d2c91db4f4d6d89983cfffa243761" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:18.954826 27275 raft_consensus.cc:457] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:18.969354 27275 raft_consensus.cc:2732] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.107s (starting election)
I0619 00:12:18.969631 27275 raft_consensus.cc:479] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.969584 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2b155ef1ed014ae5be07e75500df9659. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:18.969390 27082 raft_consensus.cc:2784] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.971876 27082 raft_consensus.cc:2732] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.599s (vote granted)
I0619 00:12:18.970721 27275 leader_election.cc:243] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.973282 27275 leader_election.cc:243] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.971158 27197 raft_consensus.cc:325] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.974970 27197 raft_consensus.cc:351] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.975177 27197 raft_consensus.cc:628] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:18.975165 26532 catalog_manager.cc:3786] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:18.967311 27200 ts_tablet_manager.cc:1382] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:18.976987 27200 log.cc:986] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:18.975752 27197 consensus_queue.cc:227] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.978668 27197 ts_tablet_manager.cc:1073] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I0619 00:12:18.978636 27200 ts_tablet_manager.cc:1403] T 1f90ed424e734a78bc242e7d4708f1f1 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:18.979025 27197 ts_tablet_manager.cc:1028] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:18.979368 27197 tablet_bootstrap.cc:427] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:18.979615 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b766f1ebdb404f5ab38c0ceedec69f37" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:18.980144 26950 raft_consensus.cc:2732] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.726s (vote granted)
I0619 00:12:18.980384 26950 raft_consensus.cc:2245] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:18.981917 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 1f90ed424e734a78bc242e7d4708f1f1 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:18.982532 27013 leader_election.cc:379] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.982797 27013 leader_election.cc:270] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:18.982970 27082 raft_consensus.cc:2245] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:18.983395 27275 raft_consensus.cc:2732] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.864s (election complete)
I0619 00:12:18.983669 27275 raft_consensus.cc:2539] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:18.983878 27275 raft_consensus.cc:457] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:18.984087 27275 raft_consensus.cc:2732] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.769s (starting election)
I0619 00:12:18.984290 27275 raft_consensus.cc:2784] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.984490 26874 leader_election.cc:379] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:18.984721 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b766f1ebdb404f5ab38c0ceedec69f37" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:18.989763 27275 raft_consensus.cc:479] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:18.986068 27015 leader_election.cc:306] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:18.990790 27275 leader_election.cc:243] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:18.991214 27275 leader_election.cc:243] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:18.991283 26532 catalog_manager.cc:3663] Got report from deleted tablet 1f90ed424e734a78bc242e7d4708f1f1 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659): Sending delete request for this tablet
I0619 00:12:18.992151 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f90ed424e734a78bc242e7d4708f1f1 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:18.993494 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b766f1ebdb404f5ab38c0ceedec69f37" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:18.994849 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 1f90ed424e734a78bc242e7d4708f1f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b155ef1ed014ae5be07e75500df9659) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:18.995538 27015 leader_election.cc:306] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:18.996115 27199 tablet_replica.cc:261] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:18.996259 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 87b2d044789641668b76263c65f8eb1a (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:18.996909 27196 tablet_bootstrap.cc:427] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:18.992969 27197 tablet_bootstrap.cc:578] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:18.992875 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b766f1ebdb404f5ab38c0ceedec69f37" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:18.997236 27196 ts_tablet_manager.cc:1045] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.078s	user 0.005s	sys 0.000s
I0619 00:12:18.997222 26950 raft_consensus.cc:2784] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:18.997469 26950 raft_consensus.cc:2732] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.649s (vote granted)
I0619 00:12:18.998549 27196 raft_consensus.cc:325] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:18.999138 27196 raft_consensus.cc:351] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:18.999307 27196 raft_consensus.cc:628] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:18.999737 27196 consensus_queue.cc:227] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.001241 27196 ts_tablet_manager.cc:1073] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:12:19.002194 27196 ts_tablet_manager.cc:1028] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.002550 27196 tablet_bootstrap.cc:427] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:19.004977 26950 raft_consensus.cc:2245] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:19.005084 27196 tablet_bootstrap.cc:578] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:18.967988 27210 consensus_peers.cc:456] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:19.007346 27210 consensus_peers.cc:456] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:19.007952 27210 raft_consensus.cc:2069] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.010279 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:19.006198 27013 leader_election.cc:379] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:19.010524 27013 leader_election.cc:270] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.011077 27275 raft_consensus.cc:2732] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.913s (election complete)
I0619 00:12:18.996896 27063 ts_tablet_manager.cc:1173] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.013173 27256 raft_consensus.cc:457] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.013351 27275 raft_consensus.cc:2539] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:19.013419 27256 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.175s (starting election)
I0619 00:12:19.013851 27275 raft_consensus.cc:593] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:19.013690 27256 raft_consensus.cc:479] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.014703 27256 leader_election.cc:243] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:19.014308 27275 consensus_queue.cc:204] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.015089 27256 leader_election.cc:243] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.016038 27195 ts_tablet_manager.cc:1028] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.016403 27195 tablet_bootstrap.cc:427] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.017586 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b155ef1ed014ae5be07e75500df9659" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:19.019928 26732 leader_election.cc:306] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:19.020624 27210 ts_tablet_manager.cc:1369] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.023000 27195 tablet_bootstrap.cc:578] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:19.023015 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b155ef1ed014ae5be07e75500df9659" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
W0619 00:12:19.024792 26732 leader_election.cc:306] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:19.025034 26732 leader_election.cc:270] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:19.025705 27256 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.144s (election complete)
I0619 00:12:19.025926 27256 raft_consensus.cc:2487] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:19.027760 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:19.028196 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:19.028435 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:19.028705 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:19.028918 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:19.029110 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(1f90ed424e734a78bc242e7d4708f1f1)
I0619 00:12:19.029302 27199 raft_consensus.cc:2042] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.029580 27199 raft_consensus.cc:2069] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.031329 27372 log.cc:525] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Log is configured to *not* fsync() on all Append() calls
I0619 00:12:19.033532 27199 ts_tablet_manager.cc:1369] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.037820 26532 catalog_manager.cc:3786] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:19.038754 27195 tablet_bootstrap.cc:427] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.039165 27195 ts_tablet_manager.cc:1045] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.005s
I0619 00:12:19.040776 27195 raft_consensus.cc:325] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.041489 27195 raft_consensus.cc:351] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.042585 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.042999 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.045398 27196 tablet_bootstrap.cc:427] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.045832 27196 ts_tablet_manager.cc:1045] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.043s	user 0.005s	sys 0.000s
I0619 00:12:19.048848 27197 tablet_bootstrap.cc:427] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:19.047282 27196 raft_consensus.cc:325] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.049262 27197 ts_tablet_manager.cc:1045] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.070s	user 0.003s	sys 0.005s
I0619 00:12:19.049198 27196 raft_consensus.cc:351] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.049494 27196 raft_consensus.cc:628] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
W0619 00:12:19.050386 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.050034 27196 consensus_queue.cc:227] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.051712 27197 raft_consensus.cc:325] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.052461 27197 raft_consensus.cc:351] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.042601 27195 raft_consensus.cc:628] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.052881 27197 raft_consensus.cc:628] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:19.053983 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.053349 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.057109 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.057240 27197 consensus_queue.cc:227] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.053514 27195 consensus_queue.cc:227] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.059154 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.056984 27196 ts_tablet_manager.cc:1073] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
W0619 00:12:19.061805 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.070708 27328 raft_consensus.cc:457] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.070942 27328 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.970s (starting election)
I0619 00:12:19.071187 27328 raft_consensus.cc:479] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.072209 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 2b155ef1ed014ae5be07e75500df9659 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:19.072597 27328 leader_election.cc:243] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.072986 27328 leader_election.cc:243] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.074492 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.075289 27197 ts_tablet_manager.cc:1073] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.001s
I0619 00:12:19.075268 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b155ef1ed014ae5be07e75500df9659" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:19.075747 26811 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.999s (vote granted)
I0619 00:12:19.075983 26811 raft_consensus.cc:2245] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.076403 27195 ts_tablet_manager.cc:1073] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.000s
I0619 00:12:19.077143 26876 leader_election.cc:379] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.079530 26876 leader_election.cc:270] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.080085 27328 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.607s (election complete)
I0619 00:12:19.080302 27328 raft_consensus.cc:2539] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.080497 27328 raft_consensus.cc:457] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.080713 27328 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.748s (starting election)
I0619 00:12:19.080899 27328 raft_consensus.cc:2784] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.081943 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b155ef1ed014ae5be07e75500df9659" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
W0619 00:12:19.082141 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.082379 27082 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.078s (vote granted)
W0619 00:12:19.082592 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.082592 27082 raft_consensus.cc:2245] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
W0619 00:12:19.084285 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.088385 26874 leader_election.cc:379] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.088872 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.088577 27328 raft_consensus.cc:479] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.093506 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 2b155ef1ed014ae5be07e75500df9659
I0619 00:12:19.093605 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.093833 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b155ef1ed014ae5be07e75500df9659 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.094223 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b155ef1ed014ae5be07e75500df9659 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.094622 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b155ef1ed014ae5be07e75500df9659 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.095705 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 2b155ef1ed014ae5be07e75500df9659 with delete_type TABLET_DATA_DELETED (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.095880 26790 tablet_service.cc:800] Processing DeleteTablet for tablet 2b155ef1ed014ae5be07e75500df9659 with delete_type TABLET_DATA_DELETED (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.095984 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 2b155ef1ed014ae5be07e75500df9659 with delete_type TABLET_DATA_DELETED (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.096365 27200 tablet_replica.cc:261] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:19.096593 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.096814 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.097000 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.097189 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.097396 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.097445 27210 ts_tablet_manager.cc:1382] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.097587 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.097669 27210 log.cc:986] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:19.097795 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.097967 27200 raft_consensus.cc:2042] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.098235 27200 raft_consensus.cc:2069] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.098266 27210 ts_tablet_manager.cc:1403] T 1f90ed424e734a78bc242e7d4708f1f1 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.099437 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 2b155ef1ed014ae5be07e75500df9659 deleted and replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0
I0619 00:12:19.099740 27200 ts_tablet_manager.cc:1369] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:19.100900 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.102288 26533 catalog_manager.cc:3663] Got report from deleted tablet 2b155ef1ed014ae5be07e75500df9659 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0): Sending delete request for this tablet
I0619 00:12:19.103387 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b155ef1ed014ae5be07e75500df9659 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.104377 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 2b155ef1ed014ae5be07e75500df9659 with delete_type TABLET_DATA_DELETED (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.104473 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet b766f1ebdb404f5ab38c0ceedec69f37 from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:19.105257 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.106484 26532 catalog_manager.cc:3663] Got report from deleted tablet 2b155ef1ed014ae5be07e75500df9659 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0): Sending delete request for this tablet
I0619 00:12:19.107126 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2b155ef1ed014ae5be07e75500df9659 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.109489 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 2b155ef1ed014ae5be07e75500df9659 with delete_type TABLET_DATA_DELETED (replaced by tablet aa0f3a5bf4204069b3ab81c5152c58f0) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.112584 27372 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=1} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0619 00:12:19.113886 26928 tablet_service.cc:759] Processing CreateTablet for tablet aa0f3a5bf4204069b3ab81c5152c58f0 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.113920 27063 tablet_service.cc:759] Processing CreateTablet for tablet aa0f3a5bf4204069b3ab81c5152c58f0 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.114462 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa0f3a5bf4204069b3ab81c5152c58f0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.114493 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa0f3a5bf4204069b3ab81c5152c58f0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.115414 27372 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Bootstrap complete.
I0619 00:12:19.117367 26790 tablet_service.cc:759] Processing CreateTablet for tablet aa0f3a5bf4204069b3ab81c5152c58f0 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.117987 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa0f3a5bf4204069b3ab81c5152c58f0. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.125279 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 1f90ed424e734a78bc242e7d4708f1f1 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:19.126355 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 1f90ed424e734a78bc242e7d4708f1f1 because the tablet was not found. No further retry: Not found: Tablet not found: 1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:19.126945 27210 tablet_replica.cc:261] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:19.127208 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.127410 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.127595 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.127826 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.128041 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.128239 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.128425 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.128592 27210 raft_consensus.cc:2042] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.128873 27210 raft_consensus.cc:2069] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.129747 27328 leader_election.cc:243] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.130910 27328 leader_election.cc:243] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.129523 26790 ts_tablet_manager.cc:1173] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.132062 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b155ef1ed014ae5be07e75500df9659" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:19.132464 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2b155ef1ed014ae5be07e75500df9659" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:19.132949 27196 ts_tablet_manager.cc:1028] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.132948 27082 raft_consensus.cc:2784] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.133180 27082 raft_consensus.cc:2732] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.625s (vote granted)
I0619 00:12:19.133334 27196 tablet_bootstrap.cc:427] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:19.136391 26876 leader_election.cc:306] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:12:19.137004 27196 tablet_bootstrap.cc:578] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:19.137739 27082 raft_consensus.cc:2245] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.138852 26874 leader_election.cc:379] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.139096 26874 leader_election.cc:270] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.139350 27199 ts_tablet_manager.cc:1382] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W0619 00:12:19.139475 26874 raft_consensus.cc:2442] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:12:19.142164 27210 ts_tablet_manager.cc:1369] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.142488 27200 ts_tablet_manager.cc:1382] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.142753 27200 log.cc:986] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/2b155ef1ed014ae5be07e75500df9659
I0619 00:12:19.143210 27200 ts_tablet_manager.cc:1403] T 2b155ef1ed014ae5be07e75500df9659 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:19.145032 27063 ts_tablet_manager.cc:1173] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.145026 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 2b155ef1ed014ae5be07e75500df9659 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:19.145535 27195 ts_tablet_manager.cc:1028] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.145931 27195 tablet_bootstrap.cc:427] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.139586 27199 log.cc:986] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:19.146076 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet b766f1ebdb404f5ab38c0ceedec69f37
I0619 00:12:19.146400 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b766f1ebdb404f5ab38c0ceedec69f37 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.146833 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b766f1ebdb404f5ab38c0ceedec69f37 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.146904 27199 ts_tablet_manager.cc:1403] T 1f90ed424e734a78bc242e7d4708f1f1 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:19.147852 26929 tablet_service.cc:800] Processing DeleteTablet for tablet b766f1ebdb404f5ab38c0ceedec69f37 with delete_type TABLET_DATA_DELETED (replaced by tablet ea1b63bf1ffa404ea936478872e775fc) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.149639 27195 tablet_bootstrap.cc:578] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:19.149906 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 1f90ed424e734a78bc242e7d4708f1f1 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:19.150924 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 1f90ed424e734a78bc242e7d4708f1f1 because the tablet was not found. No further retry: Not found: Tablet not found: 1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:19.151213 26790 tablet_service.cc:800] Processing DeleteTablet for tablet b766f1ebdb404f5ab38c0ceedec69f37 with delete_type TABLET_DATA_DELETED (replaced by tablet ea1b63bf1ffa404ea936478872e775fc) from {username='jenkins'} at 127.24.119.1:39548
W0619 00:12:19.151984 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 1f90ed424e734a78bc242e7d4708f1f1 because the tablet was not found. No further retry: Not found: Tablet not found: 1f90ed424e734a78bc242e7d4708f1f1
I0619 00:12:19.152551 27199 tablet_replica.cc:261] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:19.152833 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.153064 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.153126 26928 ts_tablet_manager.cc:1173] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.153280 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.153511 27197 ts_tablet_manager.cc:1028] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:19.153883 27197 tablet_bootstrap.cc:427] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:19.153537 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.154284 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.154520 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.154846 27200 tablet_replica.cc:261] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:19.155076 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(2b155ef1ed014ae5be07e75500df9659)
I0619 00:12:19.155283 27199 raft_consensus.cc:2042] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.155560 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.155839 27199 raft_consensus.cc:2069] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.156090 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.156357 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.156776 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.157084 27197 tablet_bootstrap.cc:578] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:19.158380 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.159076 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.159296 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.159713 27200 raft_consensus.cc:2042] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.159976 27199 ts_tablet_manager.cc:1369] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.160830 27195 tablet_bootstrap.cc:427] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.161130 27200 raft_consensus.cc:2069] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.161168 27195 ts_tablet_manager.cc:1045] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I0619 00:12:19.159829 27196 tablet_bootstrap.cc:427] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.161818 27196 ts_tablet_manager.cc:1045] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.003s
I0619 00:12:19.162704 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b766f1ebdb404f5ab38c0ceedec69f37 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.163075 26533 catalog_manager.cc:4596] ReplaceTablet: tablet b766f1ebdb404f5ab38c0ceedec69f37 deleted and replaced by tablet ea1b63bf1ffa404ea936478872e775fc
I0619 00:12:19.162669 27195 raft_consensus.cc:325] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.164247 27195 raft_consensus.cc:351] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.164503 27195 raft_consensus.cc:628] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.166568 27196 raft_consensus.cc:325] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.165899 27063 tablet_service.cc:800] Processing DeleteTablet for tablet b766f1ebdb404f5ab38c0ceedec69f37 with delete_type TABLET_DATA_DELETED (replaced by tablet ea1b63bf1ffa404ea936478872e775fc) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.176146 27062 tablet_service.cc:759] Processing CreateTablet for tablet ea1b63bf1ffa404ea936478872e775fc (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.176435 26928 tablet_service.cc:759] Processing CreateTablet for tablet ea1b63bf1ffa404ea936478872e775fc (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.176776 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ea1b63bf1ffa404ea936478872e775fc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.177037 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ea1b63bf1ffa404ea936478872e775fc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.177711 27196 raft_consensus.cc:351] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.177395 26790 tablet_service.cc:759] Processing CreateTablet for tablet ea1b63bf1ffa404ea936478872e775fc (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.178227 26790 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ea1b63bf1ffa404ea936478872e775fc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.181751 27196 raft_consensus.cc:628] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:19.182323 27196 consensus_queue.cc:227] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.184939 27197 tablet_bootstrap.cc:427] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:19.168597 27200 ts_tablet_manager.cc:1369] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.185263 27197 ts_tablet_manager.cc:1045] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.031s	user 0.003s	sys 0.002s
I0619 00:12:19.186249 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.186851 27197 raft_consensus.cc:325] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.187499 27197 raft_consensus.cc:351] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.187733 27197 raft_consensus.cc:628] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:19.188233 27197 consensus_queue.cc:227] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.190047 27197 ts_tablet_manager.cc:1073] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I0619 00:12:19.192876 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.193007 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:19.198117 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.199919 27195 consensus_queue.cc:227] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.200909 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.206323 27195 ts_tablet_manager.cc:1073] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.045s	user 0.004s	sys 0.000s
W0619 00:12:19.208714 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.208351 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.212395 27062 ts_tablet_manager.cc:1173] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.213891 26790 ts_tablet_manager.cc:1173] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:19.214265 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.215808 27195 ts_tablet_manager.cc:1028] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.215837 27196 ts_tablet_manager.cc:1073] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.054s	user 0.004s	sys 0.000s
I0619 00:12:19.216207 27195 tablet_bootstrap.cc:427] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.216202 27196 ts_tablet_manager.cc:1028] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.216536 27196 tablet_bootstrap.cc:427] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:19.217120 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.219195 27196 tablet_bootstrap.cc:578] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:19.221645 26928 ts_tablet_manager.cc:1173] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:19.221927 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.226753 27197 ts_tablet_manager.cc:1028] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:19.227118 27197 tablet_bootstrap.cc:427] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:19.229523 27197 tablet_bootstrap.cc:578] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:19.230311 27199 ts_tablet_manager.cc:1382] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.230579 27199 log.cc:986] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/2b155ef1ed014ae5be07e75500df9659
I0619 00:12:19.230741 27328 raft_consensus.cc:457] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.231016 27328 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.984s (starting election)
I0619 00:12:19.231109 27199 ts_tablet_manager.cc:1403] T 2b155ef1ed014ae5be07e75500df9659 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:19.231273 27328 raft_consensus.cc:479] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.232352 27328 leader_election.cc:243] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.232581 27200 ts_tablet_manager.cc:1382] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.232812 27328 leader_election.cc:243] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.233096 27200 log.cc:986] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/b766f1ebdb404f5ab38c0ceedec69f37
I0619 00:12:19.234375 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa0f3a5bf4204069b3ab81c5152c58f0" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:19.234870 26811 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.082s (vote granted)
I0619 00:12:19.235083 26811 raft_consensus.cc:2245] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.235672 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa0f3a5bf4204069b3ab81c5152c58f0" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:19.236179 27082 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.691s (vote granted)
I0619 00:12:19.236418 27082 raft_consensus.cc:2245] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.237736 26876 leader_election.cc:379] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.237973 26876 leader_election.cc:270] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.238404 27328 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.129s (election complete)
I0619 00:12:19.238704 26874 leader_election.cc:379] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.238683 27328 raft_consensus.cc:2539] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.239007 27328 raft_consensus.cc:457] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.239195 27328 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.931s (starting election)
I0619 00:12:19.239389 27328 raft_consensus.cc:2784] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.240186 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 2b155ef1ed014ae5be07e75500df9659 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:19.240787 27200 ts_tablet_manager.cc:1403] T b766f1ebdb404f5ab38c0ceedec69f37 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
W0619 00:12:19.241744 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 2b155ef1ed014ae5be07e75500df9659 because the tablet was not found. No further retry: Not found: Tablet not found: 2b155ef1ed014ae5be07e75500df9659
I0619 00:12:19.243590 27328 raft_consensus.cc:479] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.243935 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet b766f1ebdb404f5ab38c0ceedec69f37 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.244464 27328 leader_election.cc:243] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.244884 27328 leader_election.cc:243] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.246017 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa0f3a5bf4204069b3ab81c5152c58f0" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:19.246428 26811 raft_consensus.cc:2784] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.246660 26811 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.807s (vote granted)
I0619 00:12:19.247853 27195 tablet_bootstrap.cc:578] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:19.252199 27210 ts_tablet_manager.cc:1382] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.252454 27210 log.cc:986] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/2b155ef1ed014ae5be07e75500df9659
I0619 00:12:19.252490 26811 raft_consensus.cc:2245] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.254187 26876 leader_election.cc:379] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.254433 26876 leader_election.cc:270] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.255226 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa0f3a5bf4204069b3ab81c5152c58f0" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:19.255651 27082 raft_consensus.cc:2784] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.255904 27082 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.736s (vote granted)
I0619 00:12:19.256249 27328 raft_consensus.cc:2732] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.847s (election complete)
I0619 00:12:19.256467 27328 raft_consensus.cc:2539] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:19.257530 27328 raft_consensus.cc:593] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:19.257961 27328 consensus_queue.cc:204] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.259470 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.260540 27082 raft_consensus.cc:2245] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.261608 26874 leader_election.cc:379] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.262006 27210 ts_tablet_manager.cc:1403] T 2b155ef1ed014ae5be07e75500df9659 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.262262 27199 tablet_replica.cc:261] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:19.262751 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.263463 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 2b155ef1ed014ae5be07e75500df9659 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:19.264019 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 2b155ef1ed014ae5be07e75500df9659 because the tablet was not found. No further retry: Not found: Tablet not found: 2b155ef1ed014ae5be07e75500df9659
I0619 00:12:19.264164 27210 tablet_replica.cc:261] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:19.264389 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.263242 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.264595 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.264820 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.264849 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.265027 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.265251 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.265462 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.265280 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.265688 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.265883 27210 raft_consensus.cc:2042] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.265939 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.266157 27210 raft_consensus.cc:2069] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.266428 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.266841 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(b766f1ebdb404f5ab38c0ceedec69f37)
I0619 00:12:19.267879 27196 tablet_bootstrap.cc:427] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.268241 27196 ts_tablet_manager.cc:1045] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.052s	user 0.004s	sys 0.004s
I0619 00:12:19.269932 27196 raft_consensus.cc:325] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.270663 27196 raft_consensus.cc:351] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.270939 27199 raft_consensus.cc:2042] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:19.271159 27196 raft_consensus.cc:628] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:19.271417 27199 consensus_peers.cc:456] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:19.271656 27199 consensus_peers.cc:456] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:19.178019 27372 raft_consensus.cc:325] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:19.272532 27199 pending_rounds.cc:63] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:19.272383 27372 raft_consensus.cc:628] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8582cfc277d4d4597c5605b7ac0ed48, State: Initialized, Role: FOLLOWER
I0619 00:12:19.272210 27196 consensus_queue.cc:227] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.273385 27199 pending_rounds.cc:68] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391020609056768 op_type: NO_OP noop_request { }
I0619 00:12:19.268122 27195 tablet_bootstrap.cc:427] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.273972 27195 ts_tablet_manager.cc:1045] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.058s	user 0.004s	sys 0.001s
I0619 00:12:19.273496 27372 consensus_queue.cc:227] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:19.276041 27195 raft_consensus.cc:325] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.276639 27195 raft_consensus.cc:351] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.276865 27195 raft_consensus.cc:628] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.277391 27195 consensus_queue.cc:227] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.280607 27197 tablet_bootstrap.cc:427] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:19.281101 27197 ts_tablet_manager.cc:1045] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.054s	user 0.005s	sys 0.002s
I0619 00:12:19.281853 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.282564 27197 raft_consensus.cc:325] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.283259 27197 raft_consensus.cc:351] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.283491 27197 raft_consensus.cc:628] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:19.286252 27210 ts_tablet_manager.cc:1369] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.287853 27195 ts_tablet_manager.cc:1073] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I0619 00:12:19.284072 27197 consensus_queue.cc:227] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.291101 27197 ts_tablet_manager.cc:1073] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I0619 00:12:19.291806 26533 catalog_manager.cc:3786] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
W0619 00:12:19.293560 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.296470 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.300595 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.304913 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.305176 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet ea1b63bf1ffa404ea936478872e775fc from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:19.288574 27199 raft_consensus.cc:2622] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:19.309064 27199 raft_consensus.cc:2069] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.310981 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.314281 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.316637 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet ea1b63bf1ffa404ea936478872e775fc
W0619 00:12:19.316696 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.316920 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea1b63bf1ffa404ea936478872e775fc on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 424fa794446a444a849a34969a4e76db)
I0619 00:12:19.317301 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea1b63bf1ffa404ea936478872e775fc on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 424fa794446a444a849a34969a4e76db)
I0619 00:12:19.317701 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea1b63bf1ffa404ea936478872e775fc on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 424fa794446a444a849a34969a4e76db)
I0619 00:12:19.317848 27196 ts_tablet_manager.cc:1073] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.049s	user 0.004s	sys 0.001s
I0619 00:12:19.318022 26532 catalog_manager.cc:4596] ReplaceTablet: tablet ea1b63bf1ffa404ea936478872e775fc deleted and replaced by tablet 424fa794446a444a849a34969a4e76db
I0619 00:12:19.319033 26928 tablet_service.cc:800] Processing DeleteTablet for tablet ea1b63bf1ffa404ea936478872e775fc with delete_type TABLET_DATA_DELETED (replaced by tablet 424fa794446a444a849a34969a4e76db) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.321871 26532 catalog_manager.cc:3663] Got report from deleted tablet ea1b63bf1ffa404ea936478872e775fc (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet 424fa794446a444a849a34969a4e76db): Sending delete request for this tablet
I0619 00:12:19.322532 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea1b63bf1ffa404ea936478872e775fc on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 424fa794446a444a849a34969a4e76db)
I0619 00:12:19.323837 27062 tablet_service.cc:800] Processing DeleteTablet for tablet ea1b63bf1ffa404ea936478872e775fc with delete_type TABLET_DATA_DELETED (replaced by tablet 424fa794446a444a849a34969a4e76db) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:19.324199 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.326977 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.331596 26533 catalog_manager.cc:3663] Got report from deleted tablet ea1b63bf1ffa404ea936478872e775fc (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet 424fa794446a444a849a34969a4e76db): Sending delete request for this tablet
I0619 00:12:19.331879 26928 tablet_service.cc:759] Processing CreateTablet for tablet 424fa794446a444a849a34969a4e76db (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.332484 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 424fa794446a444a849a34969a4e76db. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:19.332556 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.321441 26790 tablet_service.cc:800] Processing DeleteTablet for tablet ea1b63bf1ffa404ea936478872e775fc with delete_type TABLET_DATA_DELETED (replaced by tablet 424fa794446a444a849a34969a4e76db) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.324102 26792 tablet_service.cc:800] Processing DeleteTablet for tablet ea1b63bf1ffa404ea936478872e775fc with delete_type TABLET_DATA_DELETED (replaced by tablet 424fa794446a444a849a34969a4e76db) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.331885 26791 tablet_service.cc:759] Processing CreateTablet for tablet 424fa794446a444a849a34969a4e76db (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.332726 27062 tablet_service.cc:759] Processing CreateTablet for tablet 424fa794446a444a849a34969a4e76db (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.333269 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 424fa794446a444a849a34969a4e76db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.333297 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 424fa794446a444a849a34969a4e76db. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:19.343966 26523 catalog_manager.cc:3083] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet ea1b63bf1ffa404ea936478872e775fc with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ea1b63bf1ffa404ea936478872e775fc already in progress: creating tablet
I0619 00:12:19.344272 26523 catalog_manager.cc:2891] Scheduling retry of ea1b63bf1ffa404ea936478872e775fc Delete Tablet RPC for TS=10c4e713d4c44dd8904e5f77ba815785 with a delay of 30 ms (attempt = 1)
W0619 00:12:19.344753 26523 catalog_manager.cc:3083] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet ea1b63bf1ffa404ea936478872e775fc with error code TABLET_NOT_RUNNING: Already present: State transition of tablet ea1b63bf1ffa404ea936478872e775fc already in progress: creating tablet
I0619 00:12:19.345082 26523 catalog_manager.cc:2891] Scheduling retry of ea1b63bf1ffa404ea936478872e775fc Delete Tablet RPC for TS=10c4e713d4c44dd8904e5f77ba815785 with a delay of 51 ms (attempt = 1)
I0619 00:12:19.346561 27199 ts_tablet_manager.cc:1369] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.344847 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea1b63bf1ffa404ea936478872e775fc on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 424fa794446a444a849a34969a4e76db)
I0619 00:12:19.350563 26929 tablet_service.cc:800] Processing DeleteTablet for tablet ea1b63bf1ffa404ea936478872e775fc with delete_type TABLET_DATA_DELETED (replaced by tablet 424fa794446a444a849a34969a4e76db) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.351023 26791 ts_tablet_manager.cc:1173] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.351830 27196 ts_tablet_manager.cc:1028] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.352191 27196 tablet_bootstrap.cc:427] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:19.353839 26928 ts_tablet_manager.cc:1173] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.354229 27197 ts_tablet_manager.cc:1028] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:19.354830 27197 tablet_bootstrap.cc:427] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:19.354854 27196 tablet_bootstrap.cc:578] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:19.356592 27372 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:19.357758 27197 tablet_bootstrap.cc:578] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:19.359313 27062 ts_tablet_manager.cc:1173] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.359732 27195 ts_tablet_manager.cc:1028] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.360097 27195 tablet_bootstrap.cc:427] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.362876 27195 tablet_bootstrap.cc:578] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:19.361697 27372 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:19.364936 27372 sys_catalog.cc:424] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:12:19.373026 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet f6f9d8a0a266441bb48b4be5a899ca5c from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:19.375025 26523 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea1b63bf1ffa404ea936478872e775fc on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 424fa794446a444a849a34969a4e76db)
I0619 00:12:19.376157 26791 tablet_service.cc:800] Processing DeleteTablet for tablet ea1b63bf1ffa404ea936478872e775fc with delete_type TABLET_DATA_DELETED (replaced by tablet 424fa794446a444a849a34969a4e76db) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.376590 27200 tablet_replica.cc:261] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:19.376839 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.377032 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.377212 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.377404 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.377619 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.377835 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.378016 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.378195 27200 raft_consensus.cc:2042] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.378468 27200 raft_consensus.cc:2069] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.380602 27200 ts_tablet_manager.cc:1369] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.386482 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet f6f9d8a0a266441bb48b4be5a899ca5c
I0619 00:12:19.386824 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f6f9d8a0a266441bb48b4be5a899ca5c on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.387249 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f6f9d8a0a266441bb48b4be5a899ca5c on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.387687 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f6f9d8a0a266441bb48b4be5a899ca5c on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.388340 26928 tablet_service.cc:800] Processing DeleteTablet for tablet f6f9d8a0a266441bb48b4be5a899ca5c with delete_type TABLET_DATA_DELETED (replaced by tablet 75e1815bfa4e43239f6364badb82b091) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.388470 26533 catalog_manager.cc:4596] ReplaceTablet: tablet f6f9d8a0a266441bb48b4be5a899ca5c deleted and replaced by tablet 75e1815bfa4e43239f6364badb82b091
I0619 00:12:19.389382 27062 tablet_service.cc:800] Processing DeleteTablet for tablet f6f9d8a0a266441bb48b4be5a899ca5c with delete_type TABLET_DATA_DELETED (replaced by tablet 75e1815bfa4e43239f6364badb82b091) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.390318 26791 tablet_service.cc:800] Processing DeleteTablet for tablet f6f9d8a0a266441bb48b4be5a899ca5c with delete_type TABLET_DATA_DELETED (replaced by tablet 75e1815bfa4e43239f6364badb82b091) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.393002 27200 ts_tablet_manager.cc:1382] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.393225 27200 log.cc:986] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/ea1b63bf1ffa404ea936478872e775fc
I0619 00:12:19.395473 27195 tablet_bootstrap.cc:427] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.396724 26523 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ea1b63bf1ffa404ea936478872e775fc on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 424fa794446a444a849a34969a4e76db)
I0619 00:12:19.398331 26791 tablet_service.cc:800] Processing DeleteTablet for tablet ea1b63bf1ffa404ea936478872e775fc with delete_type TABLET_DATA_DELETED (replaced by tablet 424fa794446a444a849a34969a4e76db) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.395815 27195 ts_tablet_manager.cc:1045] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.036s	user 0.002s	sys 0.004s
I0619 00:12:19.398746 26792 tablet_service.cc:759] Processing CreateTablet for tablet 75e1815bfa4e43239f6364badb82b091 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.399350 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 75e1815bfa4e43239f6364badb82b091. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.400009 27195 raft_consensus.cc:325] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.400590 27195 raft_consensus.cc:351] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.400790 27195 raft_consensus.cc:628] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.402055 26928 tablet_service.cc:759] Processing CreateTablet for tablet 75e1815bfa4e43239f6364badb82b091 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.402657 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 75e1815bfa4e43239f6364badb82b091. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.403587 27196 tablet_bootstrap.cc:427] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.404023 27196 ts_tablet_manager.cc:1045] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.052s	user 0.005s	sys 0.002s
I0619 00:12:19.405364 27196 raft_consensus.cc:325] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.406023 27196 raft_consensus.cc:351] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.406236 27196 raft_consensus.cc:628] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:19.406674 27196 consensus_queue.cc:227] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.408290 26792 ts_tablet_manager.cc:1173] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.413029 27210 ts_tablet_manager.cc:1382] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.413179 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.413264 27210 log.cc:986] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/b766f1ebdb404f5ab38c0ceedec69f37
I0619 00:12:19.413797 27210 ts_tablet_manager.cc:1403] T b766f1ebdb404f5ab38c0ceedec69f37 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.417959 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet b766f1ebdb404f5ab38c0ceedec69f37 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.418045 27062 tablet_service.cc:759] Processing CreateTablet for tablet 75e1815bfa4e43239f6364badb82b091 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.418695 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 75e1815bfa4e43239f6364badb82b091. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.421234 27328 raft_consensus.cc:457] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.421458 27328 raft_consensus.cc:2732] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.184s (starting election)
W0619 00:12:19.421646 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.421804 27196 ts_tablet_manager.cc:1073] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.000s
I0619 00:12:19.422148 27196 ts_tablet_manager.cc:1028] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.421730 27328 raft_consensus.cc:479] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.423008 27328 leader_election.cc:243] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.423399 27328 leader_election.cc:243] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.423511 27196 tablet_bootstrap.cc:427] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:19.423732 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.425563 27082 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ea1b63bf1ffa404ea936478872e775fc" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:19.426056 27082 raft_consensus.cc:2732] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.099s (vote granted)
I0619 00:12:19.426280 27082 raft_consensus.cc:2245] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.426908 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ea1b63bf1ffa404ea936478872e775fc" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
W0619 00:12:19.429086 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:19.429128 26876 leader_election.cc:306] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:12:19.429489 26874 leader_election.cc:379] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.429738 26874 leader_election.cc:270] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.430150 27328 raft_consensus.cc:2732] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.536s (election complete)
I0619 00:12:19.430374 27328 raft_consensus.cc:2539] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.430575 27328 raft_consensus.cc:457] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.430682 26928 ts_tablet_manager.cc:1173] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.430860 27328 raft_consensus.cc:2732] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.608s (starting election)
I0619 00:12:19.431069 27328 raft_consensus.cc:2784] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.372578 27199 ts_tablet_manager.cc:1382] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.431674 27199 log.cc:986] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/b766f1ebdb404f5ab38c0ceedec69f37
I0619 00:12:19.432324 27199 ts_tablet_manager.cc:1403] T b766f1ebdb404f5ab38c0ceedec69f37 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:19.434341 27199 tablet_replica.cc:261] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:19.434612 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.434901 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.434823 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet b766f1ebdb404f5ab38c0ceedec69f37 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.435142 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.435277 27200 ts_tablet_manager.cc:1403] T ea1b63bf1ffa404ea936478872e775fc P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:19.436128 27328 raft_consensus.cc:479] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.437031 27200 tablet_replica.cc:261] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:19.437086 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet ea1b63bf1ffa404ea936478872e775fc (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.437279 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.437530 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.437753 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.438015 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.438302 27197 tablet_bootstrap.cc:427] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:19.438247 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.438581 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.438812 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.438799 27197 ts_tablet_manager.cc:1045] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.084s	user 0.004s	sys 0.002s
I0619 00:12:19.440207 27197 raft_consensus.cc:325] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.440834 27197 raft_consensus.cc:351] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.441018 27197 raft_consensus.cc:628] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:19.441527 27197 consensus_queue.cc:227] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.445118 27197 ts_tablet_manager.cc:1073] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I0619 00:12:19.445457 27197 ts_tablet_manager.cc:1028] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:19.445809 27197 tablet_bootstrap.cc:427] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:19.446987 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.448415 27197 tablet_bootstrap.cc:578] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:19.441787 27082 raft_consensus.cc:1081] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:19.448423 27015 consensus_peers.cc:434] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Couldn't send request to peer 10c4e713d4c44dd8904e5f77ba815785 for tablet f6f9d8a0a266441bb48b4be5a899ca5c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:19.452343 27380 consensus_queue.cc:963] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:19.439173 27195 consensus_queue.cc:227] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.459998 27328 leader_election.cc:243] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.460546 27328 leader_election.cc:243] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
W0619 00:12:19.463419 26876 leader_election.cc:306] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Not found: Tablet not found: ea1b63bf1ffa404ea936478872e775fc
W0619 00:12:19.464412 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.459034 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.465178 27210 tablet_replica.cc:261] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:19.465423 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.465622 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.465827 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.466011 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.466228 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.464853 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ea1b63bf1ffa404ea936478872e775fc" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:19.466419 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.466600 27083 raft_consensus.cc:2784] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.466689 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.466856 27210 raft_consensus.cc:2042] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.466872 27083 raft_consensus.cc:2732] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.762s (vote granted)
I0619 00:12:19.467097 27210 raft_consensus.cc:2069] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.467267 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.468684 27210 ts_tablet_manager.cc:1369] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.469854 27328 raft_consensus.cc:457] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.470594 27328 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.092s (starting election)
I0619 00:12:19.470890 27328 raft_consensus.cc:479] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.471922 27328 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.472306 27195 ts_tablet_manager.cc:1073] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.074s	user 0.004s	sys 0.000s
I0619 00:12:19.472833 27328 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
W0619 00:12:19.474457 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.473428 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:19.483465 26811 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.130s (vote granted)
I0619 00:12:19.483695 26811 raft_consensus.cc:2245] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.484606 27200 raft_consensus.cc:2042] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.484841 26876 leader_election.cc:379] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.484921 27200 raft_consensus.cc:2069] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.485103 26876 leader_election.cc:270] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.488744 27328 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.238s (election complete)
I0619 00:12:19.489002 27328 raft_consensus.cc:2539] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.489270 27328 raft_consensus.cc:457] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.489483 27328 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.846s (starting election)
I0619 00:12:19.490620 27196 tablet_bootstrap.cc:578] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:19.490150 27328 raft_consensus.cc:2784] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.494263 27200 ts_tablet_manager.cc:1369] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:19.475833 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.475168 27083 raft_consensus.cc:2245] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.476510 27081 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:19.483196 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.497499 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.478219 27062 ts_tablet_manager.cc:1173] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.497741 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.497937 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(ea1b63bf1ffa404ea936478872e775fc)
I0619 00:12:19.498116 27199 raft_consensus.cc:2042] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.487835 27275 raft_consensus.cc:457] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.498211 27195 ts_tablet_manager.cc:1028] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.498389 27275 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.523s (starting election)
I0619 00:12:19.498623 27195 tablet_bootstrap.cc:427] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.498648 27275 raft_consensus.cc:479] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.499701 27275 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.500090 27275 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:19.500555 27081 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.087s (vote granted)
I0619 00:12:19.500867 27081 raft_consensus.cc:2245] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.501060 27195 tablet_bootstrap.cc:578] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:19.501842 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:19.502327 26811 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.989s (vote granted)
I0619 00:12:19.502573 26811 raft_consensus.cc:2245] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:19.503723 27015 leader_election.cc:379] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.503949 27015 leader_election.cc:270] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.504413 27275 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.709s (election complete)
I0619 00:12:19.504659 27275 raft_consensus.cc:2539] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.504861 27275 raft_consensus.cc:457] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.505059 27275 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.682s (starting election)
I0619 00:12:19.505396 27199 raft_consensus.cc:2069] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.505257 27275 raft_consensus.cc:2784] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.507614 27199 ts_tablet_manager.cc:1369] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.502696 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.511276 27275 raft_consensus.cc:479] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.512203 27275 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.513527 27275 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:19.513639 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:19.514068 26811 raft_consensus.cc:2784] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.514304 26811 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.898s (vote granted)
W0619 00:12:19.518435 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.518786 26811 raft_consensus.cc:2245] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:19.519829 27015 leader_election.cc:379] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.520053 27015 leader_election.cc:270] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.520514 27275 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.967s (election complete)
I0619 00:12:19.520119 27328 raft_consensus.cc:479] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.520803 27275 raft_consensus.cc:2539] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:19.521093 27275 raft_consensus.cc:593] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:19.518376 27195 tablet_bootstrap.cc:427] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.521898 27195 ts_tablet_manager.cc:1045] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.001s
I0619 00:12:19.522310 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:19.523088 26950 raft_consensus.cc:2175] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:19.523504 27328 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.523332 27195 raft_consensus.cc:325] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.524051 27195 raft_consensus.cc:351] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.524266 27195 raft_consensus.cc:628] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.525406 27013 leader_election.cc:393] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Invalid argument: T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:19.525504 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:19.526043 26950 raft_consensus.cc:2175] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:19.526008 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 424fa794446a444a849a34969a4e76db from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:19.523825 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.527189 27013 leader_election.cc:393] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Invalid argument: T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:19.527686 26874 leader_election.cc:379] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.527976 26874 leader_election.cc:270] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.525919 27275 consensus_queue.cc:204] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.529882 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:19.530802 26811 raft_consensus.cc:2175] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 in current term 1: Already voted for candidate a2c1dfdaa6d442539e4342d184d2167c in this term.
I0619 00:12:19.530505 27195 consensus_queue.cc:227] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.532275 27195 ts_tablet_manager.cc:1073] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I0619 00:12:19.523959 27328 leader_election.cc:243] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.535881 26876 leader_election.cc:393] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote denied by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295). Message: Invalid argument: T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 in current term 1: Already voted for candidate a2c1dfdaa6d442539e4342d184d2167c in this term.
I0619 00:12:19.536617 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 424fa794446a444a849a34969a4e76db
I0619 00:12:19.536934 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 424fa794446a444a849a34969a4e76db on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:19.537761 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 424fa794446a444a849a34969a4e76db on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:19.538127 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 424fa794446a444a849a34969a4e76db on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:19.538419 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 424fa794446a444a849a34969a4e76db deleted and replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff
I0619 00:12:19.540303 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 424fa794446a444a849a34969a4e76db with delete_type TABLET_DATA_DELETED (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.541787 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 424fa794446a444a849a34969a4e76db with delete_type TABLET_DATA_DELETED (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.540343 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.540752 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 424fa794446a444a849a34969a4e76db with delete_type TABLET_DATA_DELETED (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.547080 26532 catalog_manager.cc:3663] Got report from deleted tablet 424fa794446a444a849a34969a4e76db (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff): Sending delete request for this tablet
I0619 00:12:19.543826 26811 raft_consensus.cc:1081] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:19.548243 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 424fa794446a444a849a34969a4e76db on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:19.549530 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 424fa794446a444a849a34969a4e76db with delete_type TABLET_DATA_DELETED (replaced by tablet b7eb5e09d89b49c7abbdc8760e7ae8ff) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.553872 26792 tablet_service.cc:759] Processing CreateTablet for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.553961 27062 tablet_service.cc:759] Processing CreateTablet for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.554508 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.554512 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.557332 26928 tablet_service.cc:759] Processing CreateTablet for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.557929 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.559350 27197 tablet_bootstrap.cc:427] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
W0619 00:12:19.557518 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.562029 27197 ts_tablet_manager.cc:1045] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.116s	user 0.004s	sys 0.006s
I0619 00:12:19.563455 27197 raft_consensus.cc:325] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.566473 27199 ts_tablet_manager.cc:1382] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.564693 27403 consensus_queue.cc:963] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:19.536511 26874 raft_consensus.cc:2442] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Unable to run election callback: Service unavailable: Thread pool token was shut down
W0619 00:12:19.568482 26874 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.3:34806, user_credentials={real_user=jenkins}} blocked reactor thread for 70847.2us
I0619 00:12:19.569629 26928 ts_tablet_manager.cc:1173] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.569715 26874 leader_election.cc:379] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.572360 27200 ts_tablet_manager.cc:1382] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.572634 27200 log.cc:986] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/f6f9d8a0a266441bb48b4be5a899ca5c
W0619 00:12:19.568601 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.573859 27199 log.cc:986] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/ea1b63bf1ffa404ea936478872e775fc
I0619 00:12:19.573518 27196 tablet_bootstrap.cc:427] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.574271 27196 ts_tablet_manager.cc:1045] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.152s	user 0.005s	sys 0.003s
I0619 00:12:19.575922 27197 raft_consensus.cc:351] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.575624 27196 raft_consensus.cc:325] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.576105 27197 raft_consensus.cc:628] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:19.564509 26792 ts_tablet_manager.cc:1173] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.576489 27197 consensus_queue.cc:227] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.576545 27196 raft_consensus.cc:351] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.577347 27196 raft_consensus.cc:628] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:19.577811 27196 consensus_queue.cc:227] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.579957 27196 ts_tablet_manager.cc:1073] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I0619 00:12:19.581239 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.582636 26719 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:19.583482 27200 ts_tablet_manager.cc:1403] T f6f9d8a0a266441bb48b4be5a899ca5c P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:19.584555 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.585615 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet f6f9d8a0a266441bb48b4be5a899ca5c (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
W0619 00:12:19.586773 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet ea1b63bf1ffa404ea936478872e775fc because the tablet was not found. No further retry: Not found: Tablet not found: ea1b63bf1ffa404ea936478872e775fc
I0619 00:12:19.587476 27200 tablet_replica.cc:261] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:19.588485 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.588755 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.588989 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.589220 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.589470 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.590811 27210 ts_tablet_manager.cc:1382] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.591069 27210 log.cc:986] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/ea1b63bf1ffa404ea936478872e775fc
W0619 00:12:19.591303 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.591622 27210 ts_tablet_manager.cc:1403] T ea1b63bf1ffa404ea936478872e775fc P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.592351 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.592588 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.593485 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 75e1815bfa4e43239f6364badb82b091 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:19.593739 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.593838 27199 ts_tablet_manager.cc:1403] T ea1b63bf1ffa404ea936478872e775fc P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:19.576030 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.596674 27200 raft_consensus.cc:2042] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.597666 27197 ts_tablet_manager.cc:1073] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.000s
I0619 00:12:19.598049 27197 ts_tablet_manager.cc:1028] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
W0619 00:12:19.601786 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.572985 27062 ts_tablet_manager.cc:1173] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.602319 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "424fa794446a444a849a34969a4e76db" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:19.602885 27083 raft_consensus.cc:2206] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:19.604768 27199 tablet_replica.cc:261] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:19.605363 26874 leader_election.cc:393] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote denied by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806). Message: Invalid argument: T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:19.605790 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.605934 26874 leader_election.cc:270] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:12:19.605998 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.606192 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.606407 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.607561 27403 raft_consensus.cc:2732] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.998s (election complete)
I0619 00:12:19.607806 27403 raft_consensus.cc:2487] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:12:19.608461 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet ea1b63bf1ffa404ea936478872e775fc (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.609105 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 75e1815bfa4e43239f6364badb82b091
I0619 00:12:19.609424 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 75e1815bfa4e43239f6364badb82b091 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:19.609736 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.610023 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.609925 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 75e1815bfa4e43239f6364badb82b091 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:19.610230 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.610430 27199 raft_consensus.cc:2042] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:19.610673 27199 consensus_peers.cc:456] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:19.610894 27199 consensus_peers.cc:456] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:19.611400 27199 raft_consensus.cc:2069] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.611943 26533 catalog_manager.cc:3663] Got report from deleted tablet 75e1815bfa4e43239f6364badb82b091 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) (replaced by tablet 13d76b4f37614d938e42b87a39fc364f): Sending delete request for this tablet
I0619 00:12:19.613003 27199 ts_tablet_manager.cc:1369] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.613438 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 75e1815bfa4e43239f6364badb82b091 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:19.614117 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 75e1815bfa4e43239f6364badb82b091 with delete_type TABLET_DATA_DELETED (replaced by tablet 13d76b4f37614d938e42b87a39fc364f) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.614519 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 75e1815bfa4e43239f6364badb82b091 with delete_type TABLET_DATA_DELETED (replaced by tablet 13d76b4f37614d938e42b87a39fc364f) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.614718 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 75e1815bfa4e43239f6364badb82b091 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:19.615665 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 75e1815bfa4e43239f6364badb82b091 with delete_type TABLET_DATA_DELETED (replaced by tablet 13d76b4f37614d938e42b87a39fc364f) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:19.616235 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.616605 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 75e1815bfa4e43239f6364badb82b091 with delete_type TABLET_DATA_DELETED (replaced by tablet 13d76b4f37614d938e42b87a39fc364f) from {username='jenkins'} at 127.24.119.1:39548
W0619 00:12:19.618916 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.619549 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.622303 27195 ts_tablet_manager.cc:1028] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.622695 27195 tablet_bootstrap.cc:427] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.623366 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.625272 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.625707 27196 ts_tablet_manager.cc:1028] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.626082 27196 tablet_bootstrap.cc:427] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:19.596995 27200 raft_consensus.cc:2069] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.626734 27195 tablet_bootstrap.cc:578] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:19.629099 27196 tablet_bootstrap.cc:578] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
W0619 00:12:19.630463 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.634941 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 75e1815bfa4e43239f6364badb82b091 deleted and replaced by tablet 13d76b4f37614d938e42b87a39fc364f
I0619 00:12:19.635278 26791 tablet_service.cc:759] Processing CreateTablet for tablet 13d76b4f37614d938e42b87a39fc364f (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.635870 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 13d76b4f37614d938e42b87a39fc364f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.643859 26928 tablet_service.cc:759] Processing CreateTablet for tablet 13d76b4f37614d938e42b87a39fc364f (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.643870 27062 tablet_service.cc:759] Processing CreateTablet for tablet 13d76b4f37614d938e42b87a39fc364f (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.644428 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 13d76b4f37614d938e42b87a39fc364f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.644461 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 13d76b4f37614d938e42b87a39fc364f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.645387 27200 ts_tablet_manager.cc:1369] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.647003 27407 raft_consensus.cc:457] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.647289 27407 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.991s (starting election)
I0619 00:12:19.647572 27407 raft_consensus.cc:479] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.648641 27407 leader_election.cc:243] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.649575 27407 leader_election.cc:243] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.650166 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "75e1815bfa4e43239f6364badb82b091" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:19.650578 27197 tablet_bootstrap.cc:427] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:19.650660 26812 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.228s (vote granted)
I0619 00:12:19.650874 26812 raft_consensus.cc:2245] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.651057 27081 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "75e1815bfa4e43239f6364badb82b091" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:19.651530 27081 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.959s (vote granted)
I0619 00:12:19.651782 27081 raft_consensus.cc:2245] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.652019 26876 leader_election.cc:379] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.652312 26876 leader_election.cc:270] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.653410 26874 leader_election.cc:379] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.654374 27195 tablet_bootstrap.cc:427] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.654570 26791 ts_tablet_manager.cc:1173] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.654749 27195 ts_tablet_manager.cc:1045] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.032s	user 0.005s	sys 0.004s
I0619 00:12:19.656103 27195 raft_consensus.cc:325] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:19.656549 27403 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.757s (election complete)
I0619 00:12:19.656734 27195 raft_consensus.cc:351] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.656813 27403 raft_consensus.cc:2539] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.656951 27195 raft_consensus.cc:628] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.656991 27403 raft_consensus.cc:457] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.657169 27403 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.993s (starting election)
I0619 00:12:19.657356 27403 raft_consensus.cc:2784] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.657908 27210 tablet_replica.cc:261] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:19.657443 27195 consensus_queue.cc:227] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:19.658150 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.658951 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.659148 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.659343 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.659554 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.659771 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.659965 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(f6f9d8a0a266441bb48b4be5a899ca5c)
I0619 00:12:19.660027 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet ea1b63bf1ffa404ea936478872e775fc (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.660171 27210 raft_consensus.cc:2042] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.660459 27210 raft_consensus.cc:2069] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:19.660990 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet ea1b63bf1ffa404ea936478872e775fc because the tablet was not found. No further retry: Not found: Tablet not found: ea1b63bf1ffa404ea936478872e775fc
I0619 00:12:19.662847 27210 ts_tablet_manager.cc:1369] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.664288 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.667872 27195 ts_tablet_manager.cc:1073] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.001s
W0619 00:12:19.668668 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.670763 27196 tablet_bootstrap.cc:427] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.671115 27196 ts_tablet_manager.cc:1045] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.045s	user 0.006s	sys 0.007s
I0619 00:12:19.674149 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.674927 26811 raft_consensus.cc:1081] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:19.678336 27196 raft_consensus.cc:325] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:19.679056 27196 raft_consensus.cc:351] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:12:19.679409 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.665195 27403 raft_consensus.cc:479] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.671330 26928 ts_tablet_manager.cc:1173] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.653345 27197 tablet_bootstrap.cc:578] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:19.680166 27403 leader_election.cc:243] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.676635 27407 consensus_queue.cc:963] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:19.680981 27403 leader_election.cc:243] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.682071 27062 ts_tablet_manager.cc:1173] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.684188 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "75e1815bfa4e43239f6364badb82b091" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:19.684618 26811 raft_consensus.cc:2784] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.684870 26811 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.087s (vote granted)
I0619 00:12:19.690549 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.690830 27081 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "75e1815bfa4e43239f6364badb82b091" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:19.691267 27081 raft_consensus.cc:2784] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.691496 27081 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.820s (vote granted)
I0619 00:12:19.693198 26811 raft_consensus.cc:2245] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.694313 26876 leader_election.cc:379] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.694562 26876 leader_election.cc:270] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.695598 27081 raft_consensus.cc:2245] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.686599 27199 ts_tablet_manager.cc:1382] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.696712 26874 leader_election.cc:379] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.697021 27199 log.cc:986] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/f6f9d8a0a266441bb48b4be5a899ca5c
I0619 00:12:19.697538 27195 ts_tablet_manager.cc:1028] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.697718 27407 raft_consensus.cc:2732] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.629s (election complete)
I0619 00:12:19.697968 27407 raft_consensus.cc:2539] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:19.698124 27195 tablet_bootstrap.cc:427] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.698266 27407 raft_consensus.cc:593] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:19.698709 27407 consensus_queue.cc:204] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.700565 27195 tablet_bootstrap.cc:578] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
W0619 00:12:19.704067 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.679621 27196 raft_consensus.cc:628] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:19.709241 27197 tablet_bootstrap.cc:427] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:19.709678 27197 ts_tablet_manager.cc:1045] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.111s	user 0.006s	sys 0.005s
I0619 00:12:19.709759 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.712131 27081 raft_consensus.cc:1081] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:19.711927 27197 raft_consensus.cc:325] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:19.712644 27197 raft_consensus.cc:351] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.714134 27328 consensus_queue.cc:963] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:19.716271 26533 catalog_manager.cc:3663] Got report from deleted tablet 75e1815bfa4e43239f6364badb82b091 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) (replaced by tablet 13d76b4f37614d938e42b87a39fc364f): Sending delete request for this tablet
I0619 00:12:19.716943 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 75e1815bfa4e43239f6364badb82b091 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 13d76b4f37614d938e42b87a39fc364f)
W0619 00:12:19.718070 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.719172 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 75e1815bfa4e43239f6364badb82b091 with delete_type TABLET_DATA_DELETED (replaced by tablet 13d76b4f37614d938e42b87a39fc364f) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.720834 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.712864 27197 raft_consensus.cc:628] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:19.724138 27210 ts_tablet_manager.cc:1382] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.725971 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.728052 27199 ts_tablet_manager.cc:1403] T f6f9d8a0a266441bb48b4be5a899ca5c P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:19.729817 27199 tablet_replica.cc:261] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:19.731652 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet f6f9d8a0a266441bb48b4be5a899ca5c (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:19.725404 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.726505 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.733438 27210 log.cc:986] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/f6f9d8a0a266441bb48b4be5a899ca5c
I0619 00:12:19.723731 27197 consensus_queue.cc:227] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:19.733945 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.734028 27210 ts_tablet_manager.cc:1403] T f6f9d8a0a266441bb48b4be5a899ca5c P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.734151 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.734359 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.734571 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.735795 27210 tablet_replica.cc:261] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:19.736047 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.736043 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet f6f9d8a0a266441bb48b4be5a899ca5c (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:19.736271 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.736485 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(424fa794446a444a849a34969a4e76db)
W0619 00:12:19.737479 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:19.747223 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.749179 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 75e1815bfa4e43239f6364badb82b091 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:19.749819 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.750124 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.750376 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.750597 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.750825 27210 raft_consensus.cc:2042] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.751123 27210 raft_consensus.cc:2069] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.744329 27197 ts_tablet_manager.cc:1073] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.034s	user 0.002s	sys 0.001s
I0619 00:12:19.752593 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.753031 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.753317 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.753299 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.755769 27196 consensus_queue.cc:227] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:19.756572 27197 ts_tablet_manager.cc:1028] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:19.756824 27195 tablet_bootstrap.cc:427] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.756942 27197 tablet_bootstrap.cc:427] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:19.757184 27195 ts_tablet_manager.cc:1045] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.059s	user 0.007s	sys 0.000s
I0619 00:12:19.759299 27197 tablet_bootstrap.cc:578] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:19.760193 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(424fa794446a444a849a34969a4e76db)
I0619 00:12:19.760749 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.760823 27199 raft_consensus.cc:2042] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:19.761066 27199 consensus_peers.cc:456] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:19.761282 27199 consensus_peers.cc:456] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:19.761673 27199 pending_rounds.cc:63] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:19.761842 27199 pending_rounds.cc:68] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391022715351040 op_type: NO_OP noop_request { }
I0619 00:12:19.762825 27199 raft_consensus.cc:2622] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:19.763890 27199 raft_consensus.cc:2069] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.763047 27195 raft_consensus.cc:325] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.752569 27210 ts_tablet_manager.cc:1369] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.764860 27195 raft_consensus.cc:351] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W0619 00:12:19.765157 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.768218 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.770711 27195 raft_consensus.cc:628] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.771457 27195 consensus_queue.cc:227] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:19.772467 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.766909 27199 ts_tablet_manager.cc:1369] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.776957 27196 ts_tablet_manager.cc:1073] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.100s	user 0.004s	sys 0.000s
I0619 00:12:19.777356 27196 ts_tablet_manager.cc:1028] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.777751 27196 tablet_bootstrap.cc:427] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:19.780496 27196 tablet_bootstrap.cc:578] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:19.784601 27195 ts_tablet_manager.cc:1073] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.001s
I0619 00:12:19.784862 27200 ts_tablet_manager.cc:1382] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:12:19.785825 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.787921 27200 log.cc:986] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/424fa794446a444a849a34969a4e76db
I0619 00:12:19.788388 27200 ts_tablet_manager.cc:1403] T 424fa794446a444a849a34969a4e76db P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
W0619 00:12:19.789547 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.790050 27200 tablet_replica.cc:261] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:19.790298 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.790518 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.790750 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.790958 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.790882 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 424fa794446a444a849a34969a4e76db (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.791198 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.791383 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.791568 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.791760 27200 raft_consensus.cc:2042] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.792016 27200 raft_consensus.cc:2069] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.793233 27197 tablet_bootstrap.cc:427] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:19.793581 27197 ts_tablet_manager.cc:1045] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.037s	user 0.004s	sys 0.002s
I0619 00:12:19.794540 27200 ts_tablet_manager.cc:1369] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.795351 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.795087 27197 raft_consensus.cc:325] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.795728 27197 raft_consensus.cc:351] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.795912 27197 raft_consensus.cc:628] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:19.796329 27197 consensus_queue.cc:227] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.797829 27197 ts_tablet_manager.cc:1073] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:12:19.800195 27210 ts_tablet_manager.cc:1382] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.800463 27210 log.cc:986] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/424fa794446a444a849a34969a4e76db
I0619 00:12:19.800954 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.801003 27210 ts_tablet_manager.cc:1403] T 424fa794446a444a849a34969a4e76db P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.802836 27210 tablet_replica.cc:261] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:19.802989 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 424fa794446a444a849a34969a4e76db (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:19.806387 27199 ts_tablet_manager.cc:1382] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.802278 27196 tablet_bootstrap.cc:427] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.808909 27196 ts_tablet_manager.cc:1045] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.031s	user 0.005s	sys 0.000s
I0619 00:12:19.809530 27199 log.cc:986] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/424fa794446a444a849a34969a4e76db
I0619 00:12:19.810745 27199 ts_tablet_manager.cc:1403] T 424fa794446a444a849a34969a4e76db P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:19.811002 27196 raft_consensus.cc:325] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.812140 27196 raft_consensus.cc:351] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.812326 27196 raft_consensus.cc:628] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:19.813818 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 424fa794446a444a849a34969a4e76db (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
W0619 00:12:19.815022 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 424fa794446a444a849a34969a4e76db because the tablet was not found. No further retry: Not found: Tablet not found: 424fa794446a444a849a34969a4e76db
I0619 00:12:19.815584 27199 tablet_replica.cc:261] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:19.814463 27196 consensus_queue.cc:227] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.816280 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.816833 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.817302 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.817734 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.818461 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(75e1815bfa4e43239f6364badb82b091)
W0619 00:12:19.804956 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.820127 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.820340 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.820529 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(75e1815bfa4e43239f6364badb82b091)
W0619 00:12:19.820472 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.820741 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.820998 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.821230 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.821437 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.821642 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.821640 27210 raft_consensus.cc:2042] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:19.821913 27210 consensus_peers.cc:456] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:19.822139 27210 consensus_peers.cc:456] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:19.822516 27210 pending_rounds.cc:63] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:19.822823 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.822994 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.823196 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(75e1815bfa4e43239f6364badb82b091)
I0619 00:12:19.823374 27199 raft_consensus.cc:2042] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.823663 27199 raft_consensus.cc:2069] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.822710 27210 pending_rounds.cc:68] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391023418871808 op_type: NO_OP noop_request { }
I0619 00:12:19.824218 27210 raft_consensus.cc:2622] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:19.824470 27210 raft_consensus.cc:2069] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.824841 27196 ts_tablet_manager.cc:1073] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
I0619 00:12:19.828056 27210 ts_tablet_manager.cc:1369] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:19.828907 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.831840 27199 ts_tablet_manager.cc:1369] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.832105 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.835963 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.843024 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet aa0f3a5bf4204069b3ab81c5152c58f0 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:19.846304 27200 ts_tablet_manager.cc:1382] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.849340 27200 log.cc:986] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/75e1815bfa4e43239f6364badb82b091
I0619 00:12:19.850114 27200 ts_tablet_manager.cc:1403] T 75e1815bfa4e43239f6364badb82b091 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
W0619 00:12:19.830101 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.837188 27407 raft_consensus.cc:457] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.850826 27407 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.956s (starting election)
I0619 00:12:19.851099 27407 raft_consensus.cc:479] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.852144 27407 leader_election.cc:243] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.852548 27407 leader_election.cc:243] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.854313 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "13d76b4f37614d938e42b87a39fc364f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:19.854807 27083 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.685s (vote granted)
I0619 00:12:19.855028 27083 raft_consensus.cc:2245] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.855036 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "13d76b4f37614d938e42b87a39fc364f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:19.855743 26811 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.138s (vote granted)
I0619 00:12:19.855969 26811 raft_consensus.cc:2245] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:19.856233 26874 leader_election.cc:379] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.856273 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet aa0f3a5bf4204069b3ab81c5152c58f0
I0619 00:12:19.856472 26874 leader_election.cc:270] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.856570 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aa0f3a5bf4204069b3ab81c5152c58f0 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:19.857000 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aa0f3a5bf4204069b3ab81c5152c58f0 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:19.857241 26876 leader_election.cc:379] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.857409 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet aa0f3a5bf4204069b3ab81c5152c58f0 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:19.857774 27407 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.021s (election complete)
I0619 00:12:19.858031 27407 raft_consensus.cc:2539] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.858244 27407 raft_consensus.cc:457] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.858450 27407 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.821s (starting election)
I0619 00:12:19.858525 26791 tablet_service.cc:800] Processing DeleteTablet for tablet aa0f3a5bf4204069b3ab81c5152c58f0 with delete_type TABLET_DATA_DELETED (replaced by tablet a8af9872965f440ba4a1c7e49f18e8be) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.858659 27407 raft_consensus.cc:2784] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.861798 26532 catalog_manager.cc:4596] ReplaceTablet: tablet aa0f3a5bf4204069b3ab81c5152c58f0 deleted and replaced by tablet a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:19.862655 26928 tablet_service.cc:800] Processing DeleteTablet for tablet aa0f3a5bf4204069b3ab81c5152c58f0 with delete_type TABLET_DATA_DELETED (replaced by tablet a8af9872965f440ba4a1c7e49f18e8be) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.863582 27407 raft_consensus.cc:479] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.864557 27407 leader_election.cc:243] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.866120 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "13d76b4f37614d938e42b87a39fc364f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:19.866550 26811 raft_consensus.cc:2784] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.866811 26811 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.137s (vote granted)
I0619 00:12:19.867188 27062 tablet_service.cc:800] Processing DeleteTablet for tablet aa0f3a5bf4204069b3ab81c5152c58f0 with delete_type TABLET_DATA_DELETED (replaced by tablet a8af9872965f440ba4a1c7e49f18e8be) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.871239 26811 raft_consensus.cc:2245] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.872391 26876 leader_election.cc:379] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:19.872624 26876 leader_election.cc:270] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.873129 27328 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.741s (election complete)
I0619 00:12:19.873355 27328 raft_consensus.cc:2539] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:19.873643 27328 raft_consensus.cc:593] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:19.874101 27328 consensus_queue.cc:204] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.878268 27062 tablet_service.cc:759] Processing CreateTablet for tablet a8af9872965f440ba4a1c7e49f18e8be (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.878346 26928 tablet_service.cc:759] Processing CreateTablet for tablet a8af9872965f440ba4a1c7e49f18e8be (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.878886 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a8af9872965f440ba4a1c7e49f18e8be. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.879003 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a8af9872965f440ba4a1c7e49f18e8be. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.881152 27407 leader_election.cc:243] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.882799 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.883270 26791 tablet_service.cc:759] Processing CreateTablet for tablet a8af9872965f440ba4a1c7e49f18e8be (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.883428 27210 ts_tablet_manager.cc:1382] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.883714 27210 log.cc:986] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/75e1815bfa4e43239f6364badb82b091
I0619 00:12:19.883919 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a8af9872965f440ba4a1c7e49f18e8be. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.883729 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "13d76b4f37614d938e42b87a39fc364f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:19.884142 27083 raft_consensus.cc:2784] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.884361 27210 ts_tablet_manager.cc:1403] T 75e1815bfa4e43239f6364badb82b091 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.884357 27083 raft_consensus.cc:2732] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.599s (vote granted)
I0619 00:12:19.885474 27210 tablet_replica.cc:261] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:19.885722 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.885991 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.886167 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.886342 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.886561 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.886773 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.886878 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 75e1815bfa4e43239f6364badb82b091 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:19.887132 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(aa0f3a5bf4204069b3ab81c5152c58f0)
W0619 00:12:19.887351 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 75e1815bfa4e43239f6364badb82b091 because the tablet was not found. No further retry: Not found: Tablet not found: 75e1815bfa4e43239f6364badb82b091
I0619 00:12:19.889330 27083 raft_consensus.cc:2245] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:19.895987 27062 ts_tablet_manager.cc:1173] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.896396 27195 ts_tablet_manager.cc:1028] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:19.896745 27195 tablet_bootstrap.cc:427] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:19.896787 26791 ts_tablet_manager.cc:1173] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.897131 27196 ts_tablet_manager.cc:1028] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:19.897466 27196 tablet_bootstrap.cc:427] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:19.898994 27195 tablet_bootstrap.cc:578] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:19.899660 27196 tablet_bootstrap.cc:578] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:19.890406 26874 leader_election.cc:379] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.907129 27210 raft_consensus.cc:2042] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:19.907374 27210 consensus_peers.cc:456] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:19.907752 27210 consensus_peers.cc:456] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:19.908285 27210 raft_consensus.cc:2069] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.910861 27196 tablet_bootstrap.cc:427] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:19.911185 27196 ts_tablet_manager.cc:1045] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.002s
I0619 00:12:19.912473 27196 raft_consensus.cc:325] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.912925 27199 ts_tablet_manager.cc:1382] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:19.912994 27210 ts_tablet_manager.cc:1369] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.913177 27199 log.cc:986] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/75e1815bfa4e43239f6364badb82b091
I0619 00:12:19.913846 27196 raft_consensus.cc:351] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.914412 27196 raft_consensus.cc:628] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:19.915565 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 75e1815bfa4e43239f6364badb82b091 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
W0619 00:12:19.916347 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet 75e1815bfa4e43239f6364badb82b091 because the tablet was not found. No further retry: Not found: Tablet not found: 75e1815bfa4e43239f6364badb82b091
I0619 00:12:19.917218 27196 consensus_queue.cc:227] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.919370 27196 ts_tablet_manager.cc:1073] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I0619 00:12:19.915699 27200 tablet_replica.cc:261] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:19.902405 26928 ts_tablet_manager.cc:1173] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:19.923408 27197 ts_tablet_manager.cc:1028] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:19.923439 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.923782 27197 tablet_bootstrap.cc:427] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:19.925177 27256 raft_consensus.cc:457] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:19.925410 27256 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.883s (starting election)
I0619 00:12:19.925665 27256 raft_consensus.cc:479] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:19.926122 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.926391 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.926635 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.926862 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.928042 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.928299 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.928530 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.928772 27200 raft_consensus.cc:2042] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.929066 27200 raft_consensus.cc:2069] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.926672 27256 leader_election.cc:243] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.929605 27256 leader_election.cc:243] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:19.929934 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.931633 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b7eb5e09d89b49c7abbdc8760e7ae8ff" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:19.932107 27083 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.147s (vote granted)
I0619 00:12:19.932328 27083 raft_consensus.cc:2245] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:19.932978 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.933576 26732 leader_election.cc:379] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:19.933820 26732 leader_election.cc:270] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:19.934295 27256 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.036s (election complete)
I0619 00:12:19.934540 27256 raft_consensus.cc:2539] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:19.934749 27256 raft_consensus.cc:457] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:19.934950 27256 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.010s (starting election)
I0619 00:12:19.935142 27256 raft_consensus.cc:2784] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:19.935412 27200 ts_tablet_manager.cc:1369] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:19.937163 27195 tablet_bootstrap.cc:427] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:19.937582 27195 ts_tablet_manager.cc:1045] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.041s	user 0.006s	sys 0.002s
I0619 00:12:19.939047 27195 raft_consensus.cc:325] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.940402 27195 raft_consensus.cc:351] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:19.940888 27195 raft_consensus.cc:628] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:19.941752 27195 consensus_queue.cc:227] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:19.947122 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:19.950099 27195 ts_tablet_manager.cc:1073] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.000s
W0619 00:12:19.952356 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.954516 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:19.957507 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.959671 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet a8af9872965f440ba4a1c7e49f18e8be from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:19.960175 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.913614 27199 ts_tablet_manager.cc:1403] T 75e1815bfa4e43239f6364badb82b091 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:19.973253 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:19.973549 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a8af9872965f440ba4a1c7e49f18e8be on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:19.973976 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a8af9872965f440ba4a1c7e49f18e8be on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:19.973434 26533 catalog_manager.cc:3786] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:19.974359 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a8af9872965f440ba4a1c7e49f18e8be on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:19.975462 27199 tablet_replica.cc:261] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:19.975756 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 75e1815bfa4e43239f6364badb82b091 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:19.976565 26928 tablet_service.cc:800] Processing DeleteTablet for tablet a8af9872965f440ba4a1c7e49f18e8be with delete_type TABLET_DATA_DELETED (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:19.977520 26791 tablet_service.cc:800] Processing DeleteTablet for tablet a8af9872965f440ba4a1c7e49f18e8be with delete_type TABLET_DATA_DELETED (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:19.978266 26532 catalog_manager.cc:4596] ReplaceTablet: tablet a8af9872965f440ba4a1c7e49f18e8be deleted and replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417
I0619 00:12:19.976469 27062 tablet_service.cc:800] Processing DeleteTablet for tablet a8af9872965f440ba4a1c7e49f18e8be with delete_type TABLET_DATA_DELETED (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:19.979912 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b7eb5e09d89b49c7abbdc8760e7ae8ff" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:19.980378 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:19.980386 26950 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.580s (vote granted)
I0619 00:12:19.980515 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.980772 26950 raft_consensus.cc:2245] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:19.981137 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.981883 26732 leader_election.cc:379] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:19.982484 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.982928 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.983418 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.984266 27197 tablet_bootstrap.cc:578] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:19.984751 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.985225 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(aa0f3a5bf4204069b3ab81c5152c58f0)
I0619 00:12:19.986804 27199 raft_consensus.cc:2042] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:19.987799 27199 raft_consensus.cc:2069] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:19.996259 27062 tablet_service.cc:759] Processing CreateTablet for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.996340 26791 tablet_service.cc:759] Processing CreateTablet for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:19.996779 27210 ts_tablet_manager.cc:1382] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:19.996902 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.996946 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:19.997022 27210 log.cc:986] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/aa0f3a5bf4204069b3ab81c5152c58f0
I0619 00:12:19.967393 27256 raft_consensus.cc:479] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:20.000358 26928 tablet_service.cc:759] Processing CreateTablet for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.000982 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.001030 27256 leader_election.cc:243] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.002600 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b7eb5e09d89b49c7abbdc8760e7ae8ff" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:20.003062 27083 raft_consensus.cc:2784] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.003325 27083 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.816s (vote granted)
I0619 00:12:20.007026 26791 ts_tablet_manager.cc:1173] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.007436 27196 ts_tablet_manager.cc:1028] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:20.007827 27196 tablet_bootstrap.cc:427] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:20.007876 27083 raft_consensus.cc:2245] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:20.008972 26732 leader_election.cc:379] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.009222 26732 leader_election.cc:270] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:19.997591 27210 ts_tablet_manager.cc:1403] T aa0f3a5bf4204069b3ab81c5152c58f0 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:19.357250 27396 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:20.011723 27396 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:20.012010 27396 raft_consensus.cc:457] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.012496 27396 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.567s (starting election)
I0619 00:12:20.013093 27196 tablet_bootstrap.cc:578] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:20.013177 27062 ts_tablet_manager.cc:1173] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.013608 27195 ts_tablet_manager.cc:1028] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:20.013437 27396 raft_consensus.cc:479] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:20.014011 27195 tablet_bootstrap.cc:427] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:20.014981 27396 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:20.016243 27396 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:20.018002 26928 ts_tablet_manager.cc:1173] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.025550 27199 ts_tablet_manager.cc:1369] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.026584 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet aa0f3a5bf4204069b3ab81c5152c58f0 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:20.027289 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet a8af9872965f440ba4a1c7e49f18e8be with error code TABLET_NOT_RUNNING: Already present: State transition of tablet a8af9872965f440ba4a1c7e49f18e8be already in progress: creating tablet
I0619 00:12:20.027503 26521 catalog_manager.cc:2891] Scheduling retry of a8af9872965f440ba4a1c7e49f18e8be Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 63 ms (attempt = 1)
I0619 00:12:20.035157 27424 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.917s (election complete)
I0619 00:12:20.037602 27195 tablet_bootstrap.cc:578] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:20.038002 27424 raft_consensus.cc:2539] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:20.038133 27196 tablet_bootstrap.cc:427] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:20.039000 27196 ts_tablet_manager.cc:1045] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.031s	user 0.005s	sys 0.001s
I0619 00:12:20.008476 27256 leader_election.cc:243] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.041577 27195 tablet_bootstrap.cc:427] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:20.041676 27424 raft_consensus.cc:593] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:20.041909 27195 ts_tablet_manager.cc:1045] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.028s	user 0.005s	sys 0.002s
I0619 00:12:20.042212 27424 consensus_queue.cc:204] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:20.042883 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b7eb5e09d89b49c7abbdc8760e7ae8ff" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:20.043306 26950 raft_consensus.cc:2784] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.043546 26950 raft_consensus.cc:2732] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.975s (vote granted)
I0619 00:12:20.043267 27195 raft_consensus.cc:325] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.043731 27197 tablet_bootstrap.cc:427] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:20.043831 27195 raft_consensus.cc:351] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.044055 27197 ts_tablet_manager.cc:1045] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.120s	user 0.006s	sys 0.009s
I0619 00:12:20.044095 27195 raft_consensus.cc:628] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:20.044566 27195 consensus_queue.cc:227] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.045367 27197 raft_consensus.cc:325] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.045970 27197 raft_consensus.cc:351] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.046145 27197 raft_consensus.cc:628] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:20.046777 27197 consensus_queue.cc:227] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.050122 27197 ts_tablet_manager.cc:1073] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I0619 00:12:20.050480 27197 ts_tablet_manager.cc:1028] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:20.050866 27197 tablet_bootstrap.cc:427] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:20.050942 26533 catalog_manager.cc:3663] Got report from deleted tablet a8af9872965f440ba4a1c7e49f18e8be (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417): Sending delete request for this tablet
I0619 00:12:20.051395 27195 ts_tablet_manager.cc:1073] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I0619 00:12:20.051645 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a8af9872965f440ba4a1c7e49f18e8be on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.052752 26950 raft_consensus.cc:2245] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:20.053483 27197 tablet_bootstrap.cc:578] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:20.053598 26928 tablet_service.cc:800] Processing DeleteTablet for tablet a8af9872965f440ba4a1c7e49f18e8be with delete_type TABLET_DATA_DELETED (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.054564 27210 tablet_replica.cc:261] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:20.054844 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.055076 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.055305 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.055308 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.055553 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.055821 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.056049 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.056262 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.056464 27210 raft_consensus.cc:2042] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.056774 27210 raft_consensus.cc:2069] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.058257 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.058539 27275 raft_consensus.cc:457] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.058792 27275 raft_consensus.cc:2732] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.215s (starting election)
I0619 00:12:20.059037 27275 raft_consensus.cc:479] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.060050 27275 leader_election.cc:243] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.060469 27275 leader_election.cc:243] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.060498 26533 catalog_manager.cc:3663] Got report from deleted tablet a8af9872965f440ba4a1c7e49f18e8be (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417): Sending delete request for this tablet
I0619 00:12:20.061161 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a8af9872965f440ba4a1c7e49f18e8be on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417)
W0619 00:12:20.063720 27130 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.063982 26928 tablet_service.cc:800] Processing DeleteTablet for tablet a8af9872965f440ba4a1c7e49f18e8be with delete_type TABLET_DATA_DELETED (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.065258 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a8af9872965f440ba4a1c7e49f18e8be" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:20.065522 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.052139 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
W0619 00:12:20.066977 27013 leader_election.cc:306] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:20.063755 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a8af9872965f440ba4a1c7e49f18e8be" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:20.067859 26811 raft_consensus.cc:2732] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.763s (vote granted)
I0619 00:12:20.068073 26811 raft_consensus.cc:2245] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:20.068250 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.069506 27015 leader_election.cc:379] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.069752 27015 leader_election.cc:270] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:20.069377 27196 raft_consensus.cc:325] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.070161 27275 raft_consensus.cc:2732] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.197s (election complete)
I0619 00:12:20.070384 27275 raft_consensus.cc:2539] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:20.070595 27275 raft_consensus.cc:457] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:20.070816 27275 raft_consensus.cc:2732] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.650s (starting election)
I0619 00:12:20.064306 26732 leader_election.cc:379] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.071041 27275 raft_consensus.cc:2784] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.070025 27196 raft_consensus.cc:351] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.072094 27196 raft_consensus.cc:628] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
W0619 00:12:20.072432 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.072702 27196 consensus_queue.cc:227] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.074817 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet a8af9872965f440ba4a1c7e49f18e8be from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:20.075366 27210 ts_tablet_manager.cc:1369] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.075955 27275 raft_consensus.cc:479] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.076859 27275 leader_election.cc:243] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.077232 27275 leader_election.cc:243] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.078862 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a8af9872965f440ba4a1c7e49f18e8be" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:20.079330 26811 raft_consensus.cc:2784] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.079604 26811 raft_consensus.cc:2732] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.995s (vote granted)
I0619 00:12:20.083346 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a8af9872965f440ba4a1c7e49f18e8be" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:20.084465 26811 raft_consensus.cc:2245] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
W0619 00:12:20.086048 27013 leader_election.cc:306] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:12:20.086803 27015 leader_election.cc:379] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.087039 27015 leader_election.cc:270] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:20.088778 27275 raft_consensus.cc:2732] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.084s (election complete)
I0619 00:12:20.089385 27275 raft_consensus.cc:2539] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:20.091751 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a8af9872965f440ba4a1c7e49f18e8be on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.092752 26928 tablet_service.cc:800] Processing DeleteTablet for tablet a8af9872965f440ba4a1c7e49f18e8be with delete_type TABLET_DATA_DELETED (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.097252 26533 catalog_manager.cc:3786] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:20.098469 27199 ts_tablet_manager.cc:1382] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.098740 27199 log.cc:986] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/aa0f3a5bf4204069b3ab81c5152c58f0
I0619 00:12:20.099311 27199 ts_tablet_manager.cc:1403] T aa0f3a5bf4204069b3ab81c5152c58f0 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
W0619 00:12:20.099664 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.103667 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet aa0f3a5bf4204069b3ab81c5152c58f0 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.104048 27196 ts_tablet_manager.cc:1073] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.065s	user 0.004s	sys 0.000s
I0619 00:12:20.104796 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.105581 27199 tablet_replica.cc:261] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:20.108714 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.110205 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.109889 27275 raft_consensus.cc:593] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:20.110667 27275 consensus_queue.cc:204] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:20.111261 26856 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.112320 27435 raft_consensus.cc:457] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.112526 27197 tablet_bootstrap.cc:427] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:20.112649 27435 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.565s (starting election)
I0619 00:12:20.112960 27197 ts_tablet_manager.cc:1045] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.062s	user 0.004s	sys 0.002s
I0619 00:12:20.112938 27435 raft_consensus.cc:479] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.112932 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.113754 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.114420 27197 raft_consensus.cc:325] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.115097 27197 raft_consensus.cc:351] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.115316 27197 raft_consensus.cc:628] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:20.114017 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.116230 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.115878 27197 consensus_queue.cc:227] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.119361 27197 ts_tablet_manager.cc:1073] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I0619 00:12:20.119966 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.128505 27210 ts_tablet_manager.cc:1382] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.128914 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(a8af9872965f440ba4a1c7e49f18e8be)
W0619 00:12:20.131114 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.133872 27200 ts_tablet_manager.cc:1382] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.134131 27200 log.cc:986] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/aa0f3a5bf4204069b3ab81c5152c58f0
I0619 00:12:20.134735 27200 ts_tablet_manager.cc:1403] T aa0f3a5bf4204069b3ab81c5152c58f0 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:20.134726 27210 log.cc:986] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:20.135215 27199 raft_consensus.cc:2042] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:20.135900 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.136518 27199 consensus_peers.cc:456] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:20.136690 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet aa0f3a5bf4204069b3ab81c5152c58f0 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.137044 27199 consensus_peers.cc:456] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:20.137286 26533 catalog_manager.cc:3663] Got report from deleted tablet a8af9872965f440ba4a1c7e49f18e8be (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417): Sending delete request for this tablet
I0619 00:12:20.137727 27199 pending_rounds.cc:63] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:20.137950 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a8af9872965f440ba4a1c7e49f18e8be on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.139068 27062 tablet_service.cc:800] Processing DeleteTablet for tablet a8af9872965f440ba4a1c7e49f18e8be with delete_type TABLET_DATA_DELETED (replaced by tablet f79e7b65f2bc4b61a0bb8e7b4ee98417) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.138162 27210 ts_tablet_manager.cc:1403] T a8af9872965f440ba4a1c7e49f18e8be P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
W0619 00:12:20.140729 26996 heartbeater.cc:587] Failed to heartbeat to 127.24.119.61:43000: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.142097 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet a8af9872965f440ba4a1c7e49f18e8be (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.138244 27199 pending_rounds.cc:68] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391025100017664 op_type: NO_OP noop_request { }
I0619 00:12:20.136493 27200 tablet_replica.cc:261] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:20.144565 26678 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696" is_pre_election: true
I0619 00:12:20.145136 26678 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:20.145136 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.145431 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.114004 27435 leader_election.cc:243] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.146281 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.146517 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(a8af9872965f440ba4a1c7e49f18e8be)
W0619 00:12:20.146517 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet a8af9872965f440ba4a1c7e49f18e8be because the tablet was not found. No further retry: Not found: Tablet not found: a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:20.146836 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.146899 27435 leader_election.cc:243] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.147110 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.147521 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f79e7b65f2bc4b61a0bb8e7b4ee98417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:20.148025 26811 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.125s (vote granted)
W0619 00:12:20.148006 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet a8af9872965f440ba4a1c7e49f18e8be because the tablet was not found. No further retry: Not found: Tablet not found: a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:20.148242 26811 raft_consensus.cc:2245] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:20.148463 27199 raft_consensus.cc:2622] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:20.148432 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f79e7b65f2bc4b61a0bb8e7b4ee98417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:20.148908 26950 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.099s (vote granted)
I0619 00:12:20.149124 26950 raft_consensus.cc:2245] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:20.149608 27254 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918). Message: Invalid argument: T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
W0619 00:12:20.150063 27372 thread.cc:554] hms-catalog [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.001s	sys 0.013s
W0619 00:12:20.150257 27372 thread.cc:521] hms-catalog [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.001s	sys 0.013s
I0619 00:12:20.151172 27015 leader_election.cc:379] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.151451 27015 leader_election.cc:270] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:12:20.143275 27437 consensus_queue.cc:1042] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2c1dfdaa6d442539e4342d184d2167c" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I0619 00:12:20.153829 27013 leader_election.cc:379] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.154995 27435 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.707s (election complete)
I0619 00:12:20.155247 27435 raft_consensus.cc:2539] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:20.155458 27435 raft_consensus.cc:457] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:20.155652 27435 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.777s (starting election)
I0619 00:12:20.155880 27435 raft_consensus.cc:2784] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.157742 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(a8af9872965f440ba4a1c7e49f18e8be)
I0619 00:12:20.158114 27200 raft_consensus.cc:2042] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.159184 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:20.159773 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:20.160662 27253 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:20.160933 27253 leader_election.cc:270] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:20.164057 27130 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.164479 27200 raft_consensus.cc:2069] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.164810 27199 raft_consensus.cc:2069] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.163945 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.166610 27439 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:20.167142 27200 ts_tablet_manager.cc:1369] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.168516 27199 ts_tablet_manager.cc:1369] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.170037 27176 master_main.cc:86] Master server successfully started.
I0619 00:12:20.171296 27435 raft_consensus.cc:479] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.172349 27435 leader_election.cc:243] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.173940 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f79e7b65f2bc4b61a0bb8e7b4ee98417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:20.174417 26811 raft_consensus.cc:2784] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.174685 26811 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.920s (vote granted)
I0619 00:12:20.178746 27435 leader_election.cc:243] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.179711 26811 raft_consensus.cc:2245] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:20.180500 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f79e7b65f2bc4b61a0bb8e7b4ee98417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:20.180892 27015 leader_election.cc:379] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.180985 26950 raft_consensus.cc:2784] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.181125 27015 leader_election.cc:270] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:20.181247 26950 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.045s (vote granted)
I0619 00:12:20.183631 27435 raft_consensus.cc:2732] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.530s (election complete)
I0619 00:12:20.183910 27435 raft_consensus.cc:2539] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:20.184689 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet a8af9872965f440ba4a1c7e49f18e8be from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:20.185003 27435 raft_consensus.cc:593] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:20.185454 27435 consensus_queue.cc:204] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.186517 26950 raft_consensus.cc:2245] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:20.187832 27013 leader_election.cc:379] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.194886 27199 ts_tablet_manager.cc:1382] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.199735 27200 ts_tablet_manager.cc:1382] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.200021 27200 log.cc:986] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:20.200521 27200 ts_tablet_manager.cc:1403] T a8af9872965f440ba4a1c7e49f18e8be P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:20.201424 27199 log.cc:986] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:20.202116 27199 ts_tablet_manager.cc:1403] T a8af9872965f440ba4a1c7e49f18e8be P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:20.205927 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet a8af9872965f440ba4a1c7e49f18e8be (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:20.206706 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet a8af9872965f440ba4a1c7e49f18e8be because the tablet was not found. No further retry: Not found: Tablet not found: a8af9872965f440ba4a1c7e49f18e8be
I0619 00:12:20.206699 26533 catalog_manager.cc:3786] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:20.224509 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet a8af9872965f440ba4a1c7e49f18e8be (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.229630 27396 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.652s (election complete)
I0619 00:12:20.229882 27396 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:20.248198 27342 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.119.61:49941 (request call id 1) took 1230ms. Request Metrics: {"raft.queue_time_us":4038,"spinlock_wait_cycles":494982784,"thread_start_us":12493,"threads_started":1,"wal-append.queue_time_us":18981,"child_traces":[["txn",{"apply.queue_time_us":19131,"apply.run_cpu_time_us":1414,"apply.run_wall_time_us":2766,"num_ops":2,"prepare.queue_time_us":20485,"prepare.run_cpu_time_us":1413,"prepare.run_wall_time_us":67225,"replication_time_us":181065,"thread_start_us":44088,"threads_started":2}],["txn",{"apply.queue_time_us":3918,"apply.run_cpu_time_us":817,"apply.run_wall_time_us":6119,"num_ops":1,"prepare.queue_time_us":66208,"prepare.run_cpu_time_us":668,"prepare.run_wall_time_us":78752,"replication_time_us":148192,"row_lock_wait_count":1,"row_lock_wait_us":73622}],["txn",{"apply.queue_time_us":4672,"apply.run_cpu_time_us":813,"apply.run_wall_time_us":813,"num_ops":2,"prepare.queue_time_us":144090,"replication_time_us":147360,"row_lock_wait_count":1,"row_lock_wait_us":7742,"threads_started":1}],["txn",{"replication_time_us":147032}],["txn",{"replication_time_us":146685}],["txn",{"replication_time_us":146365}],["txn",{"replication_time_us":146068}],["txn",{"replication_time_us":145726}],["txn",{"replication_time_us":145458}],["txn",{"replication_time_us":145179}],["txn",{"replication_time_us":144862}],["txn",{"replication_time_us":144573}],["txn",{"replication_time_us":144223}],["txn",{"replication_time_us":121710}],["txn",{"replication_time_us":118509}],["txn",{"replication_time_us":117916}],["txn",{"replication_time_us":112368}],["txn",{"replication_time_us":105351}],["txn",{"replication_time_us":97583}],["txn",{"replication_time_us":88257}],["txn",{"replication_time_us":87935}],["txn",{"replication_time_us":87605}],["txn",{"replication_time_us":87322}],["txn",{"replication_time_us":75071}],["txn",{"replication_time_us":74679}],["txn",{"replication_time_us":74357}],["txn",{"replication_time_us":73689}],["txn",{"replication_time_us":73377}],["txn",{"replication_time_us":71797}],["txn",{"replication_time_us":70327}],["txn",{"replication_time_us":69981}],["txn",{"replication_time_us":69364}],["txn",{"replication_time_us":66170}],["txn",{"replication_time_us":65826}],["txn",{"replication_time_us":65475}],["txn",{"replication_time_us":65158}],["txn",{"replication_time_us":64829}],["txn",{"replication_time_us":63555}],["txn",{"replication_time_us":61515}],["txn",{"replication_time_us":60955}],["txn",{"replication_time_us":60624}],["txn",{"replication_time_us":60315}],["txn",{"replication_time_us":60002}],["txn",{"replication_time_us":58110}],["txn",{"replication_time_us":57792}],["txn",{"replication_time_us":57473}],["txn",{"replication_time_us":57150}],["txn",{"replication_time_us":54337}],["txn",{"replication_time_us":52646}],["txn",{"replication_time_us":52327}],["txn",{"replication_time_us":51979}],["txn",{"replication_time_us":51234}],["txn",{}]]}
I0619 00:12:20.253607 27418 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b46df3f26b4096876d341a82aaf7c7. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:20.254256 27418 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:20.257517 26711 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:20.258189 26711 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:20.257479 27323 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" instance_seqno: 1529392335584735) as {username='jenkins'} at 127.24.119.3:49367; Asking this server to re-register.
I0619 00:12:20.264272 27130 heartbeater.cc:426] Registering TS with master...
I0619 00:12:20.279897 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:20.287776 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 26639
I0619 00:12:20.323627 27323 ts_manager.cc:81] Registered new tserver with Master: a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.335870 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet f79e7b65f2bc4b61a0bb8e7b4ee98417
I0619 00:12:20.336187 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.336597 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.337007 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.337322 26533 catalog_manager.cc:4596] ReplaceTablet: tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 deleted and replaced by tablet 3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.337829 26928 tablet_service.cc:800] Processing DeleteTablet for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 with delete_type TABLET_DATA_DELETED (replaced by tablet 3cc1420038c54639b4346e42022dcfb2) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.338351 27062 tablet_service.cc:800] Processing DeleteTablet for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 with delete_type TABLET_DATA_DELETED (replaced by tablet 3cc1420038c54639b4346e42022dcfb2) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.338927 27210 tablet_replica.cc:261] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:20.339082 27199 tablet_replica.cc:261] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:20.339162 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339320 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339359 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339534 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339540 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339732 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339745 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339910 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.339967 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.340111 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.340159 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.340291 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.340340 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.340458 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.340519 27210 raft_consensus.cc:2042] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.340621 27199 raft_consensus.cc:2042] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:20.340831 27199 consensus_peers.cc:456] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:20.340823 27210 raft_consensus.cc:2069] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.340770 26791 tablet_service.cc:800] Processing DeleteTablet for tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 with delete_type TABLET_DATA_DELETED (replaced by tablet 3cc1420038c54639b4346e42022dcfb2) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:20.341027 27199 consensus_peers.cc:456] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:20.341267 27200 tablet_replica.cc:261] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:20.341380 27199 pending_rounds.cc:63] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:20.341509 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.341516 27199 pending_rounds.cc:68] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391025413107712 op_type: NO_OP noop_request { }
I0619 00:12:20.341732 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.341831 27199 raft_consensus.cc:2622] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:20.341913 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.342051 27199 raft_consensus.cc:2069] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.342093 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.342303 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.342494 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.342682 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(f79e7b65f2bc4b61a0bb8e7b4ee98417)
I0619 00:12:20.342849 27200 raft_consensus.cc:2042] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.343117 27200 raft_consensus.cc:2069] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.343382 27210 ts_tablet_manager.cc:1369] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.344725 27200 ts_tablet_manager.cc:1369] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.346603 27199 ts_tablet_manager.cc:1369] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.350172 27062 tablet_service.cc:759] Processing CreateTablet for tablet 3cc1420038c54639b4346e42022dcfb2 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.350790 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3cc1420038c54639b4346e42022dcfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.351335 26791 tablet_service.cc:759] Processing CreateTablet for tablet 3cc1420038c54639b4346e42022dcfb2 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.351917 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3cc1420038c54639b4346e42022dcfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.364964 26791 ts_tablet_manager.cc:1173] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.365360 27196 ts_tablet_manager.cc:1028] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:20.365725 27196 tablet_bootstrap.cc:427] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:20.368011 27196 tablet_bootstrap.cc:578] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:20.374554 26928 tablet_service.cc:759] Processing CreateTablet for tablet 3cc1420038c54639b4346e42022dcfb2 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.375273 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3cc1420038c54639b4346e42022dcfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.376323 27200 ts_tablet_manager.cc:1382] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.376567 27200 log.cc:986] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/f79e7b65f2bc4b61a0bb8e7b4ee98417
I0619 00:12:20.377049 27200 ts_tablet_manager.cc:1403] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:20.378437 27210 ts_tablet_manager.cc:1382] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.378705 27210 log.cc:986] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/f79e7b65f2bc4b61a0bb8e7b4ee98417
I0619 00:12:20.379040 26811 raft_consensus.cc:1081] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:20.379207 27210 ts_tablet_manager.cc:1403] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:20.380419 27380 consensus_queue.cc:963] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:20.380899 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.381014 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.382063 27196 tablet_bootstrap.cc:427] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:20.382406 27196 ts_tablet_manager.cc:1045] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.001s
I0619 00:12:20.382585 27062 ts_tablet_manager.cc:1173] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.382999 27195 ts_tablet_manager.cc:1028] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:20.383352 27195 tablet_bootstrap.cc:427] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:20.385408 27195 tablet_bootstrap.cc:578] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:20.385671 26856 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.389083 27196 raft_consensus.cc:325] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.389787 27196 raft_consensus.cc:351] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.391420 27196 raft_consensus.cc:628] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:20.391819 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet e90d2c91db4f4d6d89983cfffa243761 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:20.392295 27196 consensus_queue.cc:227] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.397366 27199 ts_tablet_manager.cc:1382] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.397656 27199 log.cc:986] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/f79e7b65f2bc4b61a0bb8e7b4ee98417
I0619 00:12:20.398356 27199 ts_tablet_manager.cc:1403] T f79e7b65f2bc4b61a0bb8e7b4ee98417 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:20.401335 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet e90d2c91db4f4d6d89983cfffa243761
I0619 00:12:20.401664 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e90d2c91db4f4d6d89983cfffa243761 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.402057 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e90d2c91db4f4d6d89983cfffa243761 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.402420 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e90d2c91db4f4d6d89983cfffa243761 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.402750 26533 catalog_manager.cc:4596] ReplaceTablet: tablet e90d2c91db4f4d6d89983cfffa243761 deleted and replaced by tablet 98264575a7bd4721859e79e606c98e00
I0619 00:12:20.403414 26929 tablet_service.cc:800] Processing DeleteTablet for tablet e90d2c91db4f4d6d89983cfffa243761 with delete_type TABLET_DATA_DELETED (replaced by tablet 98264575a7bd4721859e79e606c98e00) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.404739 27196 ts_tablet_manager.cc:1073] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.001s
I0619 00:12:20.406726 26791 tablet_service.cc:800] Processing DeleteTablet for tablet e90d2c91db4f4d6d89983cfffa243761 with delete_type TABLET_DATA_DELETED (replaced by tablet 98264575a7bd4721859e79e606c98e00) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:20.407217 27200 tablet_replica.cc:261] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:20.407459 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.407688 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.407882 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.408077 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.408285 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.408468 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.409703 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.412281 26929 tablet_service.cc:759] Processing CreateTablet for tablet 98264575a7bd4721859e79e606c98e00 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.412565 27200 raft_consensus.cc:2042] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.412852 27200 raft_consensus.cc:2069] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.412896 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 98264575a7bd4721859e79e606c98e00. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.413200 26791 tablet_service.cc:759] Processing CreateTablet for tablet 98264575a7bd4721859e79e606c98e00 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.413841 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 98264575a7bd4721859e79e606c98e00. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.404085 27062 tablet_service.cc:800] Processing DeleteTablet for tablet e90d2c91db4f4d6d89983cfffa243761 with delete_type TABLET_DATA_DELETED (replaced by tablet 98264575a7bd4721859e79e606c98e00) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.413076 27063 tablet_service.cc:759] Processing CreateTablet for tablet 98264575a7bd4721859e79e606c98e00 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.418880 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 98264575a7bd4721859e79e606c98e00. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.419798 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet f79e7b65f2bc4b61a0bb8e7b4ee98417 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.420399 27199 tablet_replica.cc:261] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:20.420693 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.420920 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.421141 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.421355 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.421615 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.421545 27200 ts_tablet_manager.cc:1369] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.421838 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.422047 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.422245 27199 raft_consensus.cc:2042] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.422315 26929 ts_tablet_manager.cc:1173] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.423393 27197 ts_tablet_manager.cc:1028] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:20.423759 27197 tablet_bootstrap.cc:427] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:20.425998 27197 tablet_bootstrap.cc:578] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:20.435431 27323 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" instance_seqno: 1529392333071918) as {username='jenkins'} at 127.24.119.1:48811; Asking this server to re-register.
I0619 00:12:20.438289 26856 heartbeater.cc:426] Registering TS with master...
I0619 00:12:20.439380 26996 heartbeater.cc:346] Connected to a master server at 127.24.119.61:43000
I0619 00:12:20.412263 27210 tablet_replica.cc:261] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:20.442049 27323 ts_manager.cc:81] Registered new tserver with Master: 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.443356 27199 raft_consensus.cc:2069] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.427178 27063 ts_tablet_manager.cc:1173] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.444206 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.444216 27083 raft_consensus.cc:1081] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:20.444478 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.444713 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.444923 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.445463 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.445812 27328 consensus_queue.cc:963] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:20.423044 27195 tablet_bootstrap.cc:427] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:20.451555 27195 ts_tablet_manager.cc:1045] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.068s	user 0.004s	sys 0.004s
I0619 00:12:20.445780 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.451861 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(e90d2c91db4f4d6d89983cfffa243761)
I0619 00:12:20.452085 27210 raft_consensus.cc:2042] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:20.452322 27210 consensus_peers.cc:456] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:20.452560 27210 consensus_peers.cc:456] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:20.452785 27323 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" instance_seqno: 1529392334473656) as {username='jenkins'} at 127.24.119.2:59374; Asking this server to re-register.
I0619 00:12:20.453143 27210 raft_consensus.cc:2069] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.452872 27195 raft_consensus.cc:325] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.454308 26996 heartbeater.cc:426] Registering TS with master...
I0619 00:12:20.454931 27199 ts_tablet_manager.cc:1369] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.455932 27210 ts_tablet_manager.cc:1369] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.456503 27323 ts_manager.cc:81] Registered new tserver with Master: 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.453454 27195 raft_consensus.cc:351] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.457753 27195 raft_consensus.cc:628] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:20.457965 27424 raft_consensus.cc:457] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.458209 27424 raft_consensus.cc:2732] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.103s (starting election)
I0619 00:12:20.458457 27424 raft_consensus.cc:479] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.459396 27424 leader_election.cc:243] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.459781 27424 leader_election.cc:243] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.461693 27081 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3cc1420038c54639b4346e42022dcfb2" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:20.458209 27195 consensus_queue.cc:227] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:20.462205 26732 leader_election.cc:306] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Not found: Tablet not found: 3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.462431 27081 raft_consensus.cc:2732] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.556s (vote granted)
I0619 00:12:20.462697 27081 raft_consensus.cc:2245] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:20.463716 27195 ts_tablet_manager.cc:1073] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.000s
I0619 00:12:20.464015 26732 leader_election.cc:379] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.464226 26732 leader_election.cc:270] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:20.464802 27424 raft_consensus.cc:2732] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.939s (election complete)
I0619 00:12:20.465013 27424 raft_consensus.cc:2539] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:20.465199 27424 raft_consensus.cc:457] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:20.465379 27424 raft_consensus.cc:2732] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.600s (starting election)
I0619 00:12:20.465559 27424 raft_consensus.cc:2784] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.466099 27195 ts_tablet_manager.cc:1028] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:20.466987 27195 tablet_bootstrap.cc:427] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:20.469799 27195 tablet_bootstrap.cc:578] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:20.469954 27424 raft_consensus.cc:479] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.470818 27424 leader_election.cc:243] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.471006 26791 ts_tablet_manager.cc:1173] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.471623 27424 leader_election.cc:243] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.472677 27196 ts_tablet_manager.cc:1028] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:20.473546 27196 tablet_bootstrap.cc:427] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:20.473793 26732 leader_election.cc:306] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Not found: Tablet not found: 3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.475483 27197 tablet_bootstrap.cc:427] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:20.476101 27197 ts_tablet_manager.cc:1045] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.052s	user 0.003s	sys 0.009s
I0619 00:12:20.476161 27196 tablet_bootstrap.cc:578] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:20.478106 27197 raft_consensus.cc:325] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.472218 27081 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3cc1420038c54639b4346e42022dcfb2" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:20.479144 27081 raft_consensus.cc:2784] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.479365 27081 raft_consensus.cc:2732] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.581s (vote granted)
I0619 00:12:20.483530 27081 raft_consensus.cc:2245] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:20.484694 26732 leader_election.cc:379] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.484948 26732 leader_election.cc:270] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:20.485924 27424 raft_consensus.cc:2732] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.942s (election complete)
I0619 00:12:20.486160 27424 raft_consensus.cc:2539] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:20.486443 27424 raft_consensus.cc:593] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:20.487404 27197 raft_consensus.cc:351] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.487025 27424 consensus_queue.cc:204] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.488286 27197 raft_consensus.cc:628] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:20.476516 26928 ts_tablet_manager.cc:1173] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.491952 27197 consensus_queue.cc:227] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:20.494058 26523 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918): Couldn't send request to peer 555bff44d4f942eaad13f790cca0a696 for tablet 00000000000000000000000000000000. Status: Network error: recv got EOF from 127.24.119.60:33918 (error 108). Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:20.494562 26735 connection.cc:512] client connection to 127.24.119.60:33918 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:20.495019 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:20.495213 26735 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
I0619 00:12:20.495338 27196 tablet_bootstrap.cc:427] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:20.495751 27196 ts_tablet_manager.cc:1045] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.000s
I0619 00:12:20.502324 27200 ts_tablet_manager.cc:1382] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.502564 27200 log.cc:986] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/e90d2c91db4f4d6d89983cfffa243761
W0619 00:12:20.503660 27015 connection.cc:512] client connection to 127.24.119.60:33918 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:20.504089 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:20.504940 27015 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
I0619 00:12:20.510704 27195 tablet_bootstrap.cc:427] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:20.511075 27195 ts_tablet_manager.cc:1045] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.044s	user 0.002s	sys 0.003s
I0619 00:12:20.511224 26533 catalog_manager.cc:3786] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } }
I0619 00:12:20.512615 27195 raft_consensus.cc:325] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.513303 27195 raft_consensus.cc:351] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.510359 27200 ts_tablet_manager.cc:1403] T e90d2c91db4f4d6d89983cfffa243761 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:20.513514 27195 raft_consensus.cc:628] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:20.515277 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet e90d2c91db4f4d6d89983cfffa243761 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:20.514096 27195 consensus_queue.cc:227] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:20.503324 26876 connection.cc:512] client connection to 127.24.119.60:33918 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0619 00:12:20.517771 26876 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0619 00:12:20.518224 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
I0619 00:12:20.497344 27196 raft_consensus.cc:325] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.520344 27196 raft_consensus.cc:351] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.520545 27196 raft_consensus.cc:628] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:20.521029 27196 consensus_queue.cc:227] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.522609 27196 ts_tablet_manager.cc:1073] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.027s	user 0.003s	sys 0.000s
I0619 00:12:20.534375 27197 ts_tablet_manager.cc:1073] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.057s	user 0.002s	sys 0.001s
I0619 00:12:20.537600 27197 ts_tablet_manager.cc:1028] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
W0619 00:12:20.538674 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.539227 27197 tablet_bootstrap.cc:427] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:20.541021 26950 raft_consensus.cc:1081] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:20.543653 27424 consensus_queue.cc:963] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:20.544025 27197 tablet_bootstrap.cc:578] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:20.531052 27083 raft_consensus.cc:1081] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:20.549700 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 98264575a7bd4721859e79e606c98e00 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:20.550387 27210 ts_tablet_manager.cc:1382] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.532969 27199 ts_tablet_manager.cc:1382] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.550621 27210 log.cc:986] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/e90d2c91db4f4d6d89983cfffa243761
I0619 00:12:20.550789 27199 log.cc:986] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/e90d2c91db4f4d6d89983cfffa243761
I0619 00:12:20.551230 27210 ts_tablet_manager.cc:1403] T e90d2c91db4f4d6d89983cfffa243761 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:20.551035 27424 consensus_queue.cc:963] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:20.551389 27199 ts_tablet_manager.cc:1403] T e90d2c91db4f4d6d89983cfffa243761 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:20.553093 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet e90d2c91db4f4d6d89983cfffa243761 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:20.555753 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:20.556862 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.561126 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 98264575a7bd4721859e79e606c98e00
W0619 00:12:20.561534 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.561859 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98264575a7bd4721859e79e606c98e00 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:20.562520 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98264575a7bd4721859e79e606c98e00 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:20.562845 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 98264575a7bd4721859e79e606c98e00 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.561501 26533 catalog_manager.cc:3663] Got report from deleted tablet 98264575a7bd4721859e79e606c98e00 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae): Sending delete request for this tablet
I0619 00:12:20.563462 27195 ts_tablet_manager.cc:1073] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.052s	user 0.004s	sys 0.000s
I0619 00:12:20.563532 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98264575a7bd4721859e79e606c98e00 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:20.564512 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98264575a7bd4721859e79e606c98e00 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:20.565441 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 98264575a7bd4721859e79e606c98e00 deleted and replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae
I0619 00:12:20.563709 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98264575a7bd4721859e79e606c98e00 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.565500 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 98264575a7bd4721859e79e606c98e00 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae) from {username='jenkins'} at 127.24.119.1:39548
W0619 00:12:20.564370 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.566500 27200 tablet_replica.cc:261] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:20.566764 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.566959 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.567145 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.567911 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 98264575a7bd4721859e79e606c98e00 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.567334 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.568830 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.569031 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.569209 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.569391 27200 raft_consensus.cc:2042] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.569595 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet e90d2c91db4f4d6d89983cfffa243761 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:20.569689 27200 raft_consensus.cc:2069] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:20.571079 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.571662 27199 tablet_replica.cc:261] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:20.571882 27200 ts_tablet_manager.cc:1369] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:20.572090 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.572473 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.573576 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.574146 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.575562 26928 tablet_service.cc:759] Processing CreateTablet for tablet 93d2f4cb0a4240b9a5db8ec2567993ae (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.576215 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 93d2f4cb0a4240b9a5db8ec2567993ae. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.576445 26791 tablet_service.cc:759] Processing CreateTablet for tablet 93d2f4cb0a4240b9a5db8ec2567993ae (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.577038 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 93d2f4cb0a4240b9a5db8ec2567993ae. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.578585 27435 raft_consensus.cc:457] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.578860 27435 raft_consensus.cc:2732] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.894s (starting election)
I0619 00:12:20.579277 27210 tablet_replica.cc:261] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:20.579519 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.579129 27435 raft_consensus.cc:479] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.579731 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.579905 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.580077 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.580122 27435 leader_election.cc:243] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.580281 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.580476 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.580673 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.580515 27435 leader_election.cc:243] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.585263 26791 ts_tablet_manager.cc:1173] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.586460 27196 ts_tablet_manager.cc:1028] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:20.586859 27196 tablet_bootstrap.cc:427] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:20.587349 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "98264575a7bd4721859e79e606c98e00" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:20.589354 27196 tablet_bootstrap.cc:578] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:20.590322 27062 tablet_service.cc:759] Processing CreateTablet for tablet 93d2f4cb0a4240b9a5db8ec2567993ae (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.590930 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 93d2f4cb0a4240b9a5db8ec2567993ae. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.580844 27210 raft_consensus.cc:2042] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.595928 26928 ts_tablet_manager.cc:1173] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.595870 26532 catalog_manager.cc:3663] Got report from deleted tablet 98264575a7bd4721859e79e606c98e00 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae): Sending delete request for this tablet
I0619 00:12:20.574579 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.597970 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.600441 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 98264575a7bd4721859e79e606c98e00 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:20.607770 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 3cc1420038c54639b4346e42022dcfb2 from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:20.602511 27013 leader_election.cc:306] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:20.607916 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 98264575a7bd4721859e79e606c98e00 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d2f4cb0a4240b9a5db8ec2567993ae) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.609721 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.594591 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "98264575a7bd4721859e79e606c98e00" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
W0619 00:12:20.609714 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.607375 27062 ts_tablet_manager.cc:1173] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:20.610081 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.610956 27195 ts_tablet_manager.cc:1028] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:20.611336 27195 tablet_bootstrap.cc:427] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:20.611943 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.613584 27195 tablet_bootstrap.cc:578] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
W0619 00:12:20.616214 27015 leader_election.cc:306] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: Tablet not RUNNING: STOPPED
I0619 00:12:20.616506 27015 leader_election.cc:270] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:20.616993 27435 raft_consensus.cc:2732] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.758s (election complete)
I0619 00:12:20.617209 27435 raft_consensus.cc:2487] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:20.617806 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(98264575a7bd4721859e79e606c98e00)
I0619 00:12:20.618027 27199 raft_consensus.cc:2042] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.618327 27199 raft_consensus.cc:2069] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.603214 27197 tablet_bootstrap.cc:427] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:20.621582 27197 ts_tablet_manager.cc:1045] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.083s	user 0.004s	sys 0.002s
I0619 00:12:20.623005 27197 raft_consensus.cc:325] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.623668 27197 raft_consensus.cc:351] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.623913 27197 raft_consensus.cc:628] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:20.602969 27210 raft_consensus.cc:2069] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.625643 27210 ts_tablet_manager.cc:1369] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.626524 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.626869 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3cc1420038c54639b4346e42022dcfb2 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:20.627543 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3cc1420038c54639b4346e42022dcfb2 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:20.628511 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3cc1420038c54639b4346e42022dcfb2 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:20.629439 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 3cc1420038c54639b4346e42022dcfb2 deleted and replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327
I0619 00:12:20.629500 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 3cc1420038c54639b4346e42022dcfb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327) from {username='jenkins'} at 127.24.119.1:39548
W0619 00:12:20.629699 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.631361 27199 ts_tablet_manager.cc:1369] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.631830 27196 tablet_bootstrap.cc:427] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:20.632148 27196 ts_tablet_manager.cc:1045] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.045s	user 0.005s	sys 0.003s
I0619 00:12:20.634466 27200 ts_tablet_manager.cc:1382] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.635097 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 3cc1420038c54639b4346e42022dcfb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.636849 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 3cc1420038c54639b4346e42022dcfb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.634258 27196 raft_consensus.cc:325] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.637183 27196 raft_consensus.cc:351] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.637496 27196 raft_consensus.cc:628] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:20.637464 27197 consensus_queue.cc:227] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.637967 27196 consensus_queue.cc:227] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:20.640146 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:20.641732 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:20.641832 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.642351 27197 ts_tablet_manager.cc:1073] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I0619 00:12:20.642720 27197 ts_tablet_manager.cc:1028] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:20.643085 27197 tablet_bootstrap.cc:427] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
W0619 00:12:20.645388 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.645455 27197 tablet_bootstrap.cc:578] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:20.645778 26533 catalog_manager.cc:3663] Got report from deleted tablet 3cc1420038c54639b4346e42022dcfb2 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327): Sending delete request for this tablet
I0619 00:12:20.646301 27196 ts_tablet_manager.cc:1073] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I0619 00:12:20.646441 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3cc1420038c54639b4346e42022dcfb2 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:20.648335 26791 tablet_service.cc:759] Processing CreateTablet for tablet b80e7e9623ae4a229aa5df9a9c06b327 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.648661 26928 tablet_service.cc:759] Processing CreateTablet for tablet b80e7e9623ae4a229aa5df9a9c06b327 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.648919 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b80e7e9623ae4a229aa5df9a9c06b327. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.649310 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 3cc1420038c54639b4346e42022dcfb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.649675 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b80e7e9623ae4a229aa5df9a9c06b327. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.653555 26533 catalog_manager.cc:3663] Got report from deleted tablet 3cc1420038c54639b4346e42022dcfb2 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327): Sending delete request for this tablet
I0619 00:12:20.654119 27195 tablet_bootstrap.cc:427] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:20.654204 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3cc1420038c54639b4346e42022dcfb2 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:20.654476 27195 ts_tablet_manager.cc:1045] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.043s	user 0.003s	sys 0.001s
I0619 00:12:20.655751 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 3cc1420038c54639b4346e42022dcfb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b80e7e9623ae4a229aa5df9a9c06b327) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.656970 27195 raft_consensus.cc:325] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.657574 27195 raft_consensus.cc:351] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.657789 27195 raft_consensus.cc:628] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:20.658218 27195 consensus_queue.cc:227] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.659425 27062 tablet_service.cc:759] Processing CreateTablet for tablet b80e7e9623ae4a229aa5df9a9c06b327 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.660094 27195 ts_tablet_manager.cc:1073] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
W0619 00:12:20.663051 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.665669 27210 ts_tablet_manager.cc:1382] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:12:20.666075 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.669319 26791 ts_tablet_manager.cc:1173] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.669728 27196 ts_tablet_manager.cc:1028] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:20.670050 27196 tablet_bootstrap.cc:427] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:20.637171 27200 log.cc:986] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/98264575a7bd4721859e79e606c98e00
I0619 00:12:20.671119 27200 ts_tablet_manager.cc:1403] T 98264575a7bd4721859e79e606c98e00 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:20.672065 27196 tablet_bootstrap.cc:578] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:20.673010 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 98264575a7bd4721859e79e606c98e00 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:20.673506 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b80e7e9623ae4a229aa5df9a9c06b327. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.675128 27199 ts_tablet_manager.cc:1382] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.675374 27199 log.cc:986] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/98264575a7bd4721859e79e606c98e00
I0619 00:12:20.675855 27199 ts_tablet_manager.cc:1403] T 98264575a7bd4721859e79e606c98e00 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:20.677978 27199 tablet_replica.cc:261] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:20.678613 27210 log.cc:986] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/98264575a7bd4721859e79e606c98e00
I0619 00:12:20.679145 27210 ts_tablet_manager.cc:1403] T 98264575a7bd4721859e79e606c98e00 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:20.680639 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 98264575a7bd4721859e79e606c98e00 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:20.681082 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 98264575a7bd4721859e79e606c98e00 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:20.681705 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98264575a7bd4721859e79e606c98e00 because the tablet was not found. No further retry: Not found: Tablet not found: 98264575a7bd4721859e79e606c98e00
I0619 00:12:20.682166 26928 ts_tablet_manager.cc:1173] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:20.682289 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 98264575a7bd4721859e79e606c98e00 because the tablet was not found. No further retry: Not found: Tablet not found: 98264575a7bd4721859e79e606c98e00
I0619 00:12:20.682947 27200 tablet_replica.cc:261] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:20.683457 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.683425 27197 tablet_bootstrap.cc:427] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:20.683681 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.683831 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.683864 27197 ts_tablet_manager.cc:1045] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.041s	user 0.005s	sys 0.001s
I0619 00:12:20.683892 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684075 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684098 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684298 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684495 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684341 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684717 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684855 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.684896 27200 raft_consensus.cc:2042] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:20.685098 27200 consensus_peers.cc:456] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:20.685140 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.685302 27200 consensus_peers.cc:456] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:20.685410 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.685670 27200 pending_rounds.cc:63] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:20.685667 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.683501 27210 tablet_replica.cc:261] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:20.685817 27200 pending_rounds.cc:68] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391026640633856 op_type: NO_OP noop_request { }
I0619 00:12:20.686108 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.685748 27197 raft_consensus.cc:325] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.686417 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.686502 27197 raft_consensus.cc:351] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.686777 27197 raft_consensus.cc:628] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:20.686801 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.687105 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.687506 27199 raft_consensus.cc:2042] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.687815 27199 raft_consensus.cc:2069] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.687386 27197 consensus_queue.cc:227] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:20.692207 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.692893 27200 raft_consensus.cc:2622] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:20.693503 27198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:20.694160 27198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: LEADER
W0619 00:12:20.694907 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.687350 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.698261 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.698494 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(3cc1420038c54639b4346e42022dcfb2)
I0619 00:12:20.699234 27210 raft_consensus.cc:2042] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.697551 27197 ts_tablet_manager.cc:1073] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I0619 00:12:20.697978 27199 ts_tablet_manager.cc:1369] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.700561 27200 raft_consensus.cc:2069] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.699514 27210 raft_consensus.cc:2069] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.702649 27210 ts_tablet_manager.cc:1369] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.704406 27200 ts_tablet_manager.cc:1369] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.706060 27062 ts_tablet_manager.cc:1173] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.706446 27195 ts_tablet_manager.cc:1028] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:20.706813 27195 tablet_bootstrap.cc:427] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:20.708943 27195 tablet_bootstrap.cc:578] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
W0619 00:12:20.712085 26523 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918): Couldn't send request to peer 555bff44d4f942eaad13f790cca0a696 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times.
I0619 00:12:20.720710 27196 tablet_bootstrap.cc:427] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:20.721056 27196 ts_tablet_manager.cc:1045] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.051s	user 0.002s	sys 0.008s
I0619 00:12:20.722363 27196 raft_consensus.cc:325] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.722983 27196 raft_consensus.cc:351] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.723242 27196 raft_consensus.cc:628] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:20.723695 27196 consensus_queue.cc:227] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.725877 27195 tablet_bootstrap.cc:427] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:20.726176 27195 ts_tablet_manager.cc:1045] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.019s	user 0.003s	sys 0.004s
W0619 00:12:20.728135 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.728660 27195 raft_consensus.cc:325] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.729274 27195 raft_consensus.cc:351] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.729452 27195 raft_consensus.cc:628] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:20.729935 27195 consensus_queue.cc:227] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:20.732038 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.732964 27195 ts_tablet_manager.cc:1073] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I0619 00:12:20.733779 27196 ts_tablet_manager.cc:1073] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I0619 00:12:20.734280 27197 ts_tablet_manager.cc:1028] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:20.734730 27197 tablet_bootstrap.cc:427] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
W0619 00:12:20.734844 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:20.736486 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.736927 27197 tablet_bootstrap.cc:578] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:20.743474 27199 ts_tablet_manager.cc:1382] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.743741 27199 log.cc:986] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.744199 27199 ts_tablet_manager.cc:1403] T 3cc1420038c54639b4346e42022dcfb2 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:20.745971 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 3cc1420038c54639b4346e42022dcfb2 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.749943 27200 ts_tablet_manager.cc:1382] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.750303 27200 log.cc:986] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.750109 27210 ts_tablet_manager.cc:1382] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:20.750720 27210 log.cc:986] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.751008 27200 ts_tablet_manager.cc:1403] T 3cc1420038c54639b4346e42022dcfb2 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:20.751243 27210 ts_tablet_manager.cc:1403] T 3cc1420038c54639b4346e42022dcfb2 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:20.752851 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 3cc1420038c54639b4346e42022dcfb2 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.753299 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 3cc1420038c54639b4346e42022dcfb2 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:20.753788 27197 tablet_bootstrap.cc:427] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
W0619 00:12:20.753962 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 3cc1420038c54639b4346e42022dcfb2 because the tablet was not found. No further retry: Not found: Tablet not found: 3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.754108 27197 ts_tablet_manager.cc:1045] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.002s
W0619 00:12:20.754602 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet 3cc1420038c54639b4346e42022dcfb2 because the tablet was not found. No further retry: Not found: Tablet not found: 3cc1420038c54639b4346e42022dcfb2
I0619 00:12:20.755648 27197 raft_consensus.cc:325] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.756242 27197 raft_consensus.cc:351] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.756440 27197 raft_consensus.cc:628] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:20.769956 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet b7eb5e09d89b49c7abbdc8760e7ae8ff from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:20.776819 27197 consensus_queue.cc:227] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.779451 27197 ts_tablet_manager.cc:1073] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I0619 00:12:20.781122 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet b7eb5e09d89b49c7abbdc8760e7ae8ff
I0619 00:12:20.781435 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet ca65280877d540f4aae9152095741d83)
I0619 00:12:20.782456 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet ca65280877d540f4aae9152095741d83)
I0619 00:12:20.782460 26791 tablet_service.cc:800] Processing DeleteTablet for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff with delete_type TABLET_DATA_DELETED (replaced by tablet ca65280877d540f4aae9152095741d83) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:20.783413 27200 tablet_replica.cc:261] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:20.783681 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.783879 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
W0619 00:12:20.783951 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.784065 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.784237 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.784453 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.784672 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.784855 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.785029 27200 raft_consensus.cc:2042] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:20.785225 27200 consensus_peers.cc:456] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:20.785425 27200 consensus_peers.cc:456] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:20.785951 27200 raft_consensus.cc:2069] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:20.786993 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.787997 27200 ts_tablet_manager.cc:1369] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.788417 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet ca65280877d540f4aae9152095741d83)
I0619 00:12:20.788805 26532 catalog_manager.cc:4596] ReplaceTablet: tablet b7eb5e09d89b49c7abbdc8760e7ae8ff deleted and replaced by tablet ca65280877d540f4aae9152095741d83
I0619 00:12:20.789862 26928 tablet_service.cc:800] Processing DeleteTablet for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff with delete_type TABLET_DATA_DELETED (replaced by tablet ca65280877d540f4aae9152095741d83) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:20.789863 27062 tablet_service.cc:800] Processing DeleteTablet for tablet b7eb5e09d89b49c7abbdc8760e7ae8ff with delete_type TABLET_DATA_DELETED (replaced by tablet ca65280877d540f4aae9152095741d83) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:20.790329 27210 tablet_replica.cc:261] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:20.790558 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.790788 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.790889 27199 tablet_replica.cc:261] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:20.790978 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791159 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791208 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791380 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791468 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791579 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791774 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791785 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791995 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.791956 27210 raft_consensus.cc:2042] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.792243 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.792223 27210 raft_consensus.cc:2069] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.792474 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.792713 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(b7eb5e09d89b49c7abbdc8760e7ae8ff)
I0619 00:12:20.792920 27199 raft_consensus.cc:2042] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:20.793229 27199 raft_consensus.cc:2069] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:20.795136 27210 ts_tablet_manager.cc:1369] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.796224 27199 ts_tablet_manager.cc:1369] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:20.801118 26791 tablet_service.cc:759] Processing CreateTablet for tablet ca65280877d540f4aae9152095741d83 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.801743 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ca65280877d540f4aae9152095741d83. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.801607 26928 tablet_service.cc:759] Processing CreateTablet for tablet ca65280877d540f4aae9152095741d83 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.802206 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ca65280877d540f4aae9152095741d83. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.804682 27424 raft_consensus.cc:457] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.804903 27424 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.181s (starting election)
I0619 00:12:20.805121 27424 raft_consensus.cc:479] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.806030 27424 leader_election.cc:243] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.806392 27424 leader_election.cc:243] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.807235 27210 ts_tablet_manager.cc:1382] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.807451 27210 log.cc:986] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/b7eb5e09d89b49c7abbdc8760e7ae8ff
I0619 00:12:20.807633 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "93d2f4cb0a4240b9a5db8ec2567993ae" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:20.808066 27210 ts_tablet_manager.cc:1403] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:20.808810 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "93d2f4cb0a4240b9a5db8ec2567993ae" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:20.809283 27083 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.990s (vote granted)
I0619 00:12:20.809502 27083 raft_consensus.cc:2245] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:20.809741 27062 tablet_service.cc:759] Processing CreateTablet for tablet ca65280877d540f4aae9152095741d83 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:20.810315 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ca65280877d540f4aae9152095741d83. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:20.810564 26732 leader_election.cc:379] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.810811 26732 leader_election.cc:270] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:20.811236 27424 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.560s (election complete)
I0619 00:12:20.811457 27424 raft_consensus.cc:2539] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:20.811681 27424 raft_consensus.cc:457] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:20.811872 27424 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.688s (starting election)
I0619 00:12:20.812067 27424 raft_consensus.cc:2784] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.815841 27424 raft_consensus.cc:479] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.816785 27424 leader_election.cc:243] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.817160 27424 leader_election.cc:243] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.818832 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "93d2f4cb0a4240b9a5db8ec2567993ae" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:20.819262 27083 raft_consensus.cc:2784] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.819501 27083 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.554s (vote granted)
I0619 00:12:20.822399 27199 ts_tablet_manager.cc:1382] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.822708 27199 log.cc:986] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/b7eb5e09d89b49c7abbdc8760e7ae8ff
I0619 00:12:20.823390 27199 ts_tablet_manager.cc:1403] T b7eb5e09d89b49c7abbdc8760e7ae8ff P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:20.823438 27083 raft_consensus.cc:2245] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:20.824717 26732 leader_election.cc:379] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.824762 26791 ts_tablet_manager.cc:1173] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.824982 26732 leader_election.cc:270] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:20.825247 27196 ts_tablet_manager.cc:1028] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:20.825597 27196 tablet_bootstrap.cc:427] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:20.825583 27424 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.659s (election complete)
I0619 00:12:20.825932 27424 raft_consensus.cc:2539] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:20.826216 27062 ts_tablet_manager.cc:1173] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.826997 27195 ts_tablet_manager.cc:1028] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:20.827327 27195 tablet_bootstrap.cc:427] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:20.808121 26950 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.776s (vote granted)
I0619 00:12:20.820173 26928 ts_tablet_manager.cc:1173] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:20.829341 26950 raft_consensus.cc:2245] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:20.829625 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "93d2f4cb0a4240b9a5db8ec2567993ae" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:20.830152 26949 raft_consensus.cc:2784] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.830407 26949 raft_consensus.cc:2732] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.663s (vote granted)
I0619 00:12:20.830474 27195 tablet_bootstrap.cc:578] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:20.832177 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet b7eb5e09d89b49c7abbdc8760e7ae8ff (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:20.838435 26949 raft_consensus.cc:2245] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:20.839643 27197 ts_tablet_manager.cc:1028] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:20.840037 27197 tablet_bootstrap.cc:427] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:20.842463 27197 tablet_bootstrap.cc:578] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:20.826741 27424 raft_consensus.cc:593] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:20.828236 27196 tablet_bootstrap.cc:578] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:20.844880 27424 consensus_queue.cc:204] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.834729 26732 leader_election.cc:379] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.841693 27200 ts_tablet_manager.cc:1382] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:20.846228 27200 log.cc:986] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/b7eb5e09d89b49c7abbdc8760e7ae8ff
I0619 00:12:20.846349 26732 leader_election.cc:379] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.846827 27200 ts_tablet_manager.cc:1403] T b7eb5e09d89b49c7abbdc8760e7ae8ff P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:20.848578 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet b7eb5e09d89b49c7abbdc8760e7ae8ff from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:20.828402 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.849733 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet b7eb5e09d89b49c7abbdc8760e7ae8ff (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:20.849938 27195 tablet_bootstrap.cc:427] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:20.850242 27195 ts_tablet_manager.cc:1045] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.003s
I0619 00:12:20.851462 27195 raft_consensus.cc:325] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.852068 27195 raft_consensus.cc:351] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.852253 27195 raft_consensus.cc:628] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:20.852926 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet b7eb5e09d89b49c7abbdc8760e7ae8ff (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:20.852705 27195 consensus_queue.cc:227] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.854230 27195 ts_tablet_manager.cc:1073] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
W0619 00:12:20.855034 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:20.861076 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.862644 27197 tablet_bootstrap.cc:427] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:20.863088 27197 ts_tablet_manager.cc:1045] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.023s	user 0.003s	sys 0.006s
I0619 00:12:20.864642 27197 raft_consensus.cc:325] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.865311 27197 raft_consensus.cc:351] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.865528 27197 raft_consensus.cc:628] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:20.866103 27197 consensus_queue.cc:227] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.871686 27196 tablet_bootstrap.cc:427] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:20.872018 27196 ts_tablet_manager.cc:1045] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.047s	user 0.006s	sys 0.005s
W0619 00:12:20.872323 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.873672 27196 raft_consensus.cc:325] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.874277 27196 raft_consensus.cc:351] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:20.874460 27196 raft_consensus.cc:628] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:20.874900 27196 consensus_queue.cc:227] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.875418 26532 catalog_manager.cc:3786] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:20.876778 27196 ts_tablet_manager.cc:1073] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:12:20.878083 27435 raft_consensus.cc:457] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.878350 27435 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.029s (starting election)
I0619 00:12:20.878614 27435 raft_consensus.cc:479] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:20.879489 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.879590 27435 leader_election.cc:243] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.880004 27435 leader_election.cc:243] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.882462 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b80e7e9623ae4a229aa5df9a9c06b327" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:20.882772 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b80e7e9623ae4a229aa5df9a9c06b327" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:20.882997 26949 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.965s (vote granted)
I0619 00:12:20.883227 26811 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.227s (vote granted)
I0619 00:12:20.883213 26949 raft_consensus.cc:2245] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:20.883442 26811 raft_consensus.cc:2245] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
W0619 00:12:20.884115 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.885182 27013 leader_election.cc:379] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:20.885396 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.885409 27013 leader_election.cc:270] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:20.885864 27435 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.187s (election complete)
I0619 00:12:20.886083 27435 raft_consensus.cc:2539] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:20.886320 27435 raft_consensus.cc:457] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:20.886517 27435 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.109s (starting election)
I0619 00:12:20.886736 27435 raft_consensus.cc:2784] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.886195 27015 leader_election.cc:379] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.897146 27197 ts_tablet_manager.cc:1073] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.034s	user 0.004s	sys 0.000s
I0619 00:12:20.896314 27435 raft_consensus.cc:479] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.899303 27435 leader_election.cc:243] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.900749 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b80e7e9623ae4a229aa5df9a9c06b327" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:20.901175 26949 raft_consensus.cc:2784] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.901422 26949 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.050s (vote granted)
I0619 00:12:20.901687 27435 leader_election.cc:243] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.905244 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b80e7e9623ae4a229aa5df9a9c06b327" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:20.905694 26811 raft_consensus.cc:2784] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.905670 26949 raft_consensus.cc:2245] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:20.905937 26811 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.562s (vote granted)
I0619 00:12:20.908237 27013 leader_election.cc:379] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.908468 27013 leader_election.cc:270] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:20.909090 27435 raft_consensus.cc:2732] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.783s (election complete)
I0619 00:12:20.909343 27435 raft_consensus.cc:2539] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:20.910609 26811 raft_consensus.cc:2245] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:20.911797 27015 leader_election.cc:379] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:20.916759 27435 raft_consensus.cc:593] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
W0619 00:12:20.918465 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:20.918351 27435 consensus_queue.cc:204] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:20.928249 26532 catalog_manager.cc:3786] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:20.951244 25052 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.60:33918
--webserver_interface=127.24.119.60
--webserver_port=32921
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--hive_metastore_notification_log_poll_period_seconds=1
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_sasl_enabled=false with env {}
I0619 00:12:20.980363 27424 raft_consensus.cc:457] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:20.980645 27424 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.099s (starting election)
I0619 00:12:20.980922 27424 raft_consensus.cc:479] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.981997 27424 leader_election.cc:243] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.982429 27424 leader_election.cc:243] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.983976 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca65280877d540f4aae9152095741d83" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:20.984087 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca65280877d540f4aae9152095741d83" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:20.984446 27083 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.982s (vote granted)
I0619 00:12:20.984557 26949 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.963s (vote granted)
I0619 00:12:20.984688 27083 raft_consensus.cc:2245] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:20.984815 26949 raft_consensus.cc:2245] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:20.986052 26732 leader_election.cc:379] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.986279 26732 leader_election.cc:270] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:20.986752 27424 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.566s (election complete)
I0619 00:12:20.986901 26732 leader_election.cc:379] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.986981 27424 raft_consensus.cc:2539] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:20.987205 27424 raft_consensus.cc:457] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:20.987409 27424 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.633s (starting election)
I0619 00:12:20.987620 27424 raft_consensus.cc:2784] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.991968 27424 raft_consensus.cc:479] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:20.992868 27424 leader_election.cc:243] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:20.993250 27424 leader_election.cc:243] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:20.994341 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca65280877d540f4aae9152095741d83" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:20.994793 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca65280877d540f4aae9152095741d83" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:20.995141 26949 raft_consensus.cc:2784] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.995214 27083 raft_consensus.cc:2784] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:20.995343 26949 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.180s (vote granted)
I0619 00:12:20.995451 27083 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.784s (vote granted)
I0619 00:12:21.000051 27083 raft_consensus.cc:2245] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:21.001094 26732 leader_election.cc:379] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.000013 26949 raft_consensus.cc:2245] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:21.001324 26732 leader_election.cc:270] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:21.002061 27424 raft_consensus.cc:2732] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.575s (election complete)
I0619 00:12:21.002264 26732 leader_election.cc:379] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.002285 27424 raft_consensus.cc:2539] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:21.003433 27424 raft_consensus.cc:593] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:21.004796 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 13d76b4f37614d938e42b87a39fc364f from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:21.004856 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.004317 27424 consensus_queue.cc:204] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.011103 26533 catalog_manager.cc:3786] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:21.015255 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 13d76b4f37614d938e42b87a39fc364f
I0619 00:12:21.015475 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 13d76b4f37614d938e42b87a39fc364f on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.015808 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 13d76b4f37614d938e42b87a39fc364f on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.016185 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 13d76b4f37614d938e42b87a39fc364f on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.016393 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 13d76b4f37614d938e42b87a39fc364f with delete_type TABLET_DATA_DELETED (replaced by tablet e719602f08ff4deab45c4ae9a231fe94) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.016499 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 13d76b4f37614d938e42b87a39fc364f deleted and replaced by tablet e719602f08ff4deab45c4ae9a231fe94
I0619 00:12:21.016872 27200 tablet_replica.cc:261] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.017105 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.017307 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.017491 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.017699 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.017918 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.017913 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 13d76b4f37614d938e42b87a39fc364f with delete_type TABLET_DATA_DELETED (replaced by tablet e719602f08ff4deab45c4ae9a231fe94) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.018108 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.018287 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.018405 27210 tablet_replica.cc:261] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.018458 27200 raft_consensus.cc:2042] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.018631 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.018738 27200 raft_consensus.cc:2069] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.018853 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.019031 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.019210 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.019421 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.019486 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 13d76b4f37614d938e42b87a39fc364f with delete_type TABLET_DATA_DELETED (replaced by tablet e719602f08ff4deab45c4ae9a231fe94) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.019611 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.019814 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.019991 27199 tablet_replica.cc:261] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.019991 27210 raft_consensus.cc:2042] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:21.020185 27210 consensus_peers.cc:456] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:21.020211 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.020396 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.020390 27210 consensus_peers.cc:456] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:21.020573 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.020774 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.020920 27210 raft_consensus.cc:2069] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.020998 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.021180 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.021359 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(13d76b4f37614d938e42b87a39fc364f)
I0619 00:12:21.021524 27199 raft_consensus.cc:2042] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.021806 27199 raft_consensus.cc:2069] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.023123 27200 ts_tablet_manager.cc:1369] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.025648 27199 ts_tablet_manager.cc:1369] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.027034 27210 ts_tablet_manager.cc:1369] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.041548 26928 tablet_service.cc:759] Processing CreateTablet for tablet e719602f08ff4deab45c4ae9a231fe94 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.042191 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e719602f08ff4deab45c4ae9a231fe94. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.042989 26791 tablet_service.cc:759] Processing CreateTablet for tablet e719602f08ff4deab45c4ae9a231fe94 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.043571 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e719602f08ff4deab45c4ae9a231fe94. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.046309 27062 tablet_service.cc:759] Processing CreateTablet for tablet e719602f08ff4deab45c4ae9a231fe94 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.046939 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e719602f08ff4deab45c4ae9a231fe94. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.054350 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 13d76b4f37614d938e42b87a39fc364f from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.057945 27200 ts_tablet_manager.cc:1382] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.058523 27210 ts_tablet_manager.cc:1382] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.058843 27200 log.cc:986] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/13d76b4f37614d938e42b87a39fc364f
I0619 00:12:21.059506 27200 ts_tablet_manager.cc:1403] T 13d76b4f37614d938e42b87a39fc364f P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:21.059754 27210 log.cc:986] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/13d76b4f37614d938e42b87a39fc364f
I0619 00:12:21.060463 27210 ts_tablet_manager.cc:1403] T 13d76b4f37614d938e42b87a39fc364f P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.061533 27062 ts_tablet_manager.cc:1173] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.057607 26928 ts_tablet_manager.cc:1173] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.061961 27195 ts_tablet_manager.cc:1028] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:21.062333 27195 tablet_bootstrap.cc:427] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:21.062609 27197 ts_tablet_manager.cc:1028] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:21.062963 27197 tablet_bootstrap.cc:427] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:21.065232 27197 tablet_bootstrap.cc:578] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:21.067754 27195 tablet_bootstrap.cc:578] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:21.070261 26791 ts_tablet_manager.cc:1173] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.070431 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 13d76b4f37614d938e42b87a39fc364f (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:21.071713 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 13d76b4f37614d938e42b87a39fc364f (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:21.073985 27196 ts_tablet_manager.cc:1028] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:21.074335 27196 tablet_bootstrap.cc:427] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:21.076253 27199 ts_tablet_manager.cc:1382] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.076485 27199 log.cc:986] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/13d76b4f37614d938e42b87a39fc364f
I0619 00:12:21.076668 27196 tablet_bootstrap.cc:578] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:21.077033 27199 ts_tablet_manager.cc:1403] T 13d76b4f37614d938e42b87a39fc364f P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:21.080253 27196 tablet_bootstrap.cc:427] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:21.080526 27196 ts_tablet_manager.cc:1045] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I0619 00:12:21.081742 27196 raft_consensus.cc:325] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.082304 27196 raft_consensus.cc:351] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.082468 27196 raft_consensus.cc:628] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:21.083493 27196 consensus_queue.cc:227] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.086390 27196 ts_tablet_manager.cc:1073] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
W0619 00:12:21.087414 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.088307 27195 tablet_bootstrap.cc:427] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:21.088646 27195 ts_tablet_manager.cc:1045] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.026s	user 0.004s	sys 0.002s
I0619 00:12:21.090082 27195 raft_consensus.cc:325] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.090711 27195 raft_consensus.cc:351] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.090900 27195 raft_consensus.cc:628] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:21.091338 27195 consensus_queue.cc:227] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.092032 27197 tablet_bootstrap.cc:427] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:21.093917 27195 ts_tablet_manager.cc:1073] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:12:21.100571 27197 ts_tablet_manager.cc:1045] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.002s
I0619 00:12:21.102294 27197 raft_consensus.cc:325] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.103040 27197 raft_consensus.cc:351] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.103257 27197 raft_consensus.cc:628] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:21.103921 27197 consensus_queue.cc:227] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.110730 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.116441 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.117383 27197 ts_tablet_manager.cc:1073] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
W0619 00:12:21.094441 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.119585 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.120838 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 13d76b4f37614d938e42b87a39fc364f (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
W0619 00:12:21.128914 26523 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918): Couldn't send request to peer 555bff44d4f942eaad13f790cca0a696 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times.
I0619 00:12:21.148586 27435 raft_consensus.cc:457] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:21.148954 27435 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.911s (starting election)
I0619 00:12:21.149229 27435 raft_consensus.cc:479] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.150255 27435 leader_election.cc:243] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.150717 27435 leader_election.cc:243] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.152573 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e719602f08ff4deab45c4ae9a231fe94" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:21.153059 26949 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.103s (vote granted)
I0619 00:12:21.153228 26949 raft_consensus.cc:2245] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:21.153167 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e719602f08ff4deab45c4ae9a231fe94" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:21.153698 26811 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.096s (vote granted)
I0619 00:12:21.153921 26811 raft_consensus.cc:2245] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:21.154333 27013 leader_election.cc:379] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.154539 27013 leader_election.cc:270] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:21.155174 27015 leader_election.cc:379] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.155681 27435 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.586s (election complete)
I0619 00:12:21.155930 27435 raft_consensus.cc:2539] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:21.156131 27435 raft_consensus.cc:457] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:21.156318 27435 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.850s (starting election)
I0619 00:12:21.156507 27435 raft_consensus.cc:2784] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.161115 27435 raft_consensus.cc:479] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.162699 27435 leader_election.cc:243] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.163130 27435 leader_election.cc:243] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.164713 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e719602f08ff4deab45c4ae9a231fe94" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:21.165150 26949 raft_consensus.cc:2784] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.165393 26949 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.092s (vote granted)
I0619 00:12:21.165251 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e719602f08ff4deab45c4ae9a231fe94" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:21.165699 26811 raft_consensus.cc:2784] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.165940 26811 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.147s (vote granted)
I0619 00:12:21.166072 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:21.170337 26811 raft_consensus.cc:2245] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:21.171392 27015 leader_election.cc:379] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.171658 27015 leader_election.cc:270] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:21.172149 27435 raft_consensus.cc:2732] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.859s (election complete)
I0619 00:12:21.172382 27435 raft_consensus.cc:2539] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:21.172382 26949 raft_consensus.cc:2245] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:21.172696 27435 raft_consensus.cc:593] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:21.173475 27013 leader_election.cc:379] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.173127 27435 consensus_queue.cc:204] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.177098 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.184913 26533 catalog_manager.cc:3786] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:21.213328 27439 catalog_manager.cc:1062] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: acquiring CA information for follower catalog manager: success
I0619 00:12:21.227687 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 93d2f4cb0a4240b9a5db8ec2567993ae from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.238173 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 93d2f4cb0a4240b9a5db8ec2567993ae
I0619 00:12:21.238482 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93d2f4cb0a4240b9a5db8ec2567993ae on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.238894 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93d2f4cb0a4240b9a5db8ec2567993ae on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.239517 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 93d2f4cb0a4240b9a5db8ec2567993ae with delete_type TABLET_DATA_DELETED (replaced by tablet 87f31dc0733b4059aa0d5d85999d1ea5) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.240051 27210 tablet_replica.cc:261] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.240278 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.240286 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 93d2f4cb0a4240b9a5db8ec2567993ae with delete_type TABLET_DATA_DELETED (replaced by tablet 87f31dc0733b4059aa0d5d85999d1ea5) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.240471 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.240650 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.240773 27200 tablet_replica.cc:261] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.240862 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.240994 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241077 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241188 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241261 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241364 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241436 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241542 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241614 27210 raft_consensus.cc:2042] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.241775 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.241900 27210 raft_consensus.cc:2069] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.241960 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.242157 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.242326 27200 raft_consensus.cc:2042] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:21.242519 27200 consensus_peers.cc:456] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:21.242738 27200 consensus_peers.cc:456] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:21.243064 27200 pending_rounds.cc:63] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:21.243175 27210 ts_tablet_manager.cc:1369] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.243204 27200 pending_rounds.cc:68] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391028118020096 op_type: NO_OP noop_request { }
I0619 00:12:21.243494 27200 raft_consensus.cc:2622] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:21.244163 27200 raft_consensus.cc:2069] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.244668 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 93d2f4cb0a4240b9a5db8ec2567993ae on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.247118 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 93d2f4cb0a4240b9a5db8ec2567993ae with delete_type TABLET_DATA_DELETED (replaced by tablet 87f31dc0733b4059aa0d5d85999d1ea5) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.247586 27199 tablet_replica.cc:261] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.247838 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.248023 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.248193 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.248373 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.248584 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.248802 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.248991 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(93d2f4cb0a4240b9a5db8ec2567993ae)
I0619 00:12:21.249162 27199 raft_consensus.cc:2042] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.249426 27199 raft_consensus.cc:2069] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.250744 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 93d2f4cb0a4240b9a5db8ec2567993ae deleted and replaced by tablet 87f31dc0733b4059aa0d5d85999d1ea5
I0619 00:12:21.250602 27199 ts_tablet_manager.cc:1369] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.258513 27200 ts_tablet_manager.cc:1369] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.262756 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet b80e7e9623ae4a229aa5df9a9c06b327 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.263931 27439 catalog_manager.cc:1090] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0619 00:12:21.266973 26928 tablet_service.cc:759] Processing CreateTablet for tablet 87f31dc0733b4059aa0d5d85999d1ea5 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.267058 26791 tablet_service.cc:759] Processing CreateTablet for tablet 87f31dc0733b4059aa0d5d85999d1ea5 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.267593 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 87f31dc0733b4059aa0d5d85999d1ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.267639 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 87f31dc0733b4059aa0d5d85999d1ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.271028 27062 tablet_service.cc:759] Processing CreateTablet for tablet 87f31dc0733b4059aa0d5d85999d1ea5 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.274832 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 87f31dc0733b4059aa0d5d85999d1ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.280130 26928 ts_tablet_manager.cc:1173] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.286917 27062 ts_tablet_manager.cc:1173] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.287309 27195 ts_tablet_manager.cc:1028] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:21.287672 27195 tablet_bootstrap.cc:427] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:21.288098 27210 ts_tablet_manager.cc:1382] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.288342 27210 log.cc:986] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/93d2f4cb0a4240b9a5db8ec2567993ae
I0619 00:12:21.288856 27210 ts_tablet_manager.cc:1403] T 93d2f4cb0a4240b9a5db8ec2567993ae P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.289921 27195 tablet_bootstrap.cc:578] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:21.291172 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 93d2f4cb0a4240b9a5db8ec2567993ae (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:21.294203 26791 ts_tablet_manager.cc:1173] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.296612 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet b80e7e9623ae4a229aa5df9a9c06b327
I0619 00:12:21.297003 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b80e7e9623ae4a229aa5df9a9c06b327 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.297726 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b80e7e9623ae4a229aa5df9a9c06b327 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.298079 26929 tablet_service.cc:800] Processing DeleteTablet for tablet b80e7e9623ae4a229aa5df9a9c06b327 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ec58633baa442bb7edbddc1655bd38) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.298936 27199 ts_tablet_manager.cc:1382] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.299101 27063 tablet_service.cc:800] Processing DeleteTablet for tablet b80e7e9623ae4a229aa5df9a9c06b327 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ec58633baa442bb7edbddc1655bd38) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.299187 27199 log.cc:986] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/93d2f4cb0a4240b9a5db8ec2567993ae
I0619 00:12:21.299594 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b80e7e9623ae4a229aa5df9a9c06b327 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.299726 27199 ts_tablet_manager.cc:1403] T 93d2f4cb0a4240b9a5db8ec2567993ae P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:21.301496 27199 tablet_replica.cc:261] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.301491 26792 tablet_service.cc:800] Processing DeleteTablet for tablet b80e7e9623ae4a229aa5df9a9c06b327 with delete_type TABLET_DATA_DELETED (replaced by tablet 59ec58633baa442bb7edbddc1655bd38) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.301702 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 93d2f4cb0a4240b9a5db8ec2567993ae (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:21.301880 27210 tablet_replica.cc:261] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.302098 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.302281 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.302444 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.302467 27195 tablet_bootstrap.cc:427] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:21.302608 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.302811 27195 ts_tablet_manager.cc:1045] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.001s
I0619 00:12:21.302836 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.303071 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.303257 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.303433 27210 raft_consensus.cc:2042] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.303714 27210 raft_consensus.cc:2069] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.304124 26533 catalog_manager.cc:4596] ReplaceTablet: tablet b80e7e9623ae4a229aa5df9a9c06b327 deleted and replaced by tablet 59ec58633baa442bb7edbddc1655bd38
I0619 00:12:21.304208 27195 raft_consensus.cc:325] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.304870 27195 raft_consensus.cc:351] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.305083 27195 raft_consensus.cc:628] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:21.305166 27210 ts_tablet_manager.cc:1369] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.305701 27195 consensus_queue.cc:227] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.310550 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.312662 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.316735 27197 ts_tablet_manager.cc:1028] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:21.317111 27197 tablet_bootstrap.cc:427] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:21.317277 26929 tablet_service.cc:759] Processing CreateTablet for tablet 59ec58633baa442bb7edbddc1655bd38 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.317873 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 59ec58633baa442bb7edbddc1655bd38. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.317991 26792 tablet_service.cc:759] Processing CreateTablet for tablet 59ec58633baa442bb7edbddc1655bd38 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.318574 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 59ec58633baa442bb7edbddc1655bd38. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.319916 27196 ts_tablet_manager.cc:1028] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:21.320303 27196 tablet_bootstrap.cc:427] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:21.321648 27197 tablet_bootstrap.cc:578] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:21.323456 27196 tablet_bootstrap.cc:578] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:21.325069 27062 tablet_service.cc:759] Processing CreateTablet for tablet 59ec58633baa442bb7edbddc1655bd38 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.325645 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 59ec58633baa442bb7edbddc1655bd38. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.330793 27200 ts_tablet_manager.cc:1382] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.332429 26929 ts_tablet_manager.cc:1173] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.333269 27200 log.cc:986] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/93d2f4cb0a4240b9a5db8ec2567993ae
I0619 00:12:21.334067 27200 ts_tablet_manager.cc:1403] T 93d2f4cb0a4240b9a5db8ec2567993ae P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:21.335675 26792 ts_tablet_manager.cc:1173] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.337791 27196 tablet_bootstrap.cc:427] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:21.338045 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.338284 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.338490 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.338706 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.338930 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.338729 27196 ts_tablet_manager.cc:1045] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.003s
I0619 00:12:21.339216 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.339402 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.339584 27199 raft_consensus.cc:2042] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:21.339746 27200 tablet_replica.cc:261] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.339828 27199 consensus_peers.cc:456] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:21.339901 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 93d2f4cb0a4240b9a5db8ec2567993ae (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:21.340034 27199 consensus_peers.cc:456] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:21.340032 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.340385 27199 pending_rounds.cc:63] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:21.340608 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.340531 27199 pending_rounds.cc:68] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391028407906304 op_type: NO_OP noop_request { }
I0619 00:12:21.338901 27195 ts_tablet_manager.cc:1073] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.000s
I0619 00:12:21.341899 27197 tablet_bootstrap.cc:427] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:21.342255 27197 ts_tablet_manager.cc:1045] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.006s
I0619 00:12:21.343595 27197 raft_consensus.cc:325] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.344256 27197 raft_consensus.cc:351] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.344473 27197 raft_consensus.cc:628] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:21.345043 27197 consensus_queue.cc:227] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.349202 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.352399 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.353236 27197 ts_tablet_manager.cc:1073] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.000s
I0619 00:12:21.353636 27197 ts_tablet_manager.cc:1028] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:21.354526 27197 tablet_bootstrap.cc:427] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:21.357161 27197 tablet_bootstrap.cc:578] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:21.340256 27196 raft_consensus.cc:325] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.357424 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.357455 27196 raft_consensus.cc:351] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.357794 27196 raft_consensus.cc:628] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:21.358232 27196 consensus_queue.cc:227] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.360066 27196 ts_tablet_manager.cc:1073] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I0619 00:12:21.360396 27196 ts_tablet_manager.cc:1028] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:21.360772 27196 tablet_bootstrap.cc:427] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:21.364970 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.366087 27062 ts_tablet_manager.cc:1173] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.357755 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.340852 27199 raft_consensus.cc:2622] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:21.366842 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.366973 27199 raft_consensus.cc:2069] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.367066 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.367274 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(b80e7e9623ae4a229aa5df9a9c06b327)
I0619 00:12:21.366664 27195 ts_tablet_manager.cc:1028] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:21.367961 27195 tablet_bootstrap.cc:427] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:21.367466 27200 raft_consensus.cc:2042] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.368342 27200 raft_consensus.cc:2069] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:21.368454 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.369280 27199 ts_tablet_manager.cc:1369] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.370687 27200 ts_tablet_manager.cc:1369] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.371887 27196 tablet_bootstrap.cc:578] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:21.375216 27195 tablet_bootstrap.cc:578] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:21.383539 27196 tablet_bootstrap.cc:427] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:21.384016 27196 ts_tablet_manager.cc:1045] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.001s
I0619 00:12:21.385525 27196 raft_consensus.cc:325] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.388480 27210 ts_tablet_manager.cc:1382] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.388764 27210 log.cc:986] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/b80e7e9623ae4a229aa5df9a9c06b327
I0619 00:12:21.389230 27210 ts_tablet_manager.cc:1403] T b80e7e9623ae4a229aa5df9a9c06b327 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.389430 27196 raft_consensus.cc:351] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.389853 27196 raft_consensus.cc:628] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:21.390473 27197 tablet_bootstrap.cc:427] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:21.390813 27197 ts_tablet_manager.cc:1045] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.037s	user 0.003s	sys 0.001s
I0619 00:12:21.391185 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet b80e7e9623ae4a229aa5df9a9c06b327 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:21.392117 27197 raft_consensus.cc:325] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.392733 27197 raft_consensus.cc:351] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.392698 27435 raft_consensus.cc:457] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:21.392961 27435 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.683s (starting election)
I0619 00:12:21.392949 27197 raft_consensus.cc:628] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:21.393205 27435 raft_consensus.cc:479] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.393494 27197 consensus_queue.cc:227] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.394234 27435 leader_election.cc:243] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.394650 27435 leader_election.cc:243] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.390542 27196 consensus_queue.cc:227] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.396075 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.399128 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.399929 27197 ts_tablet_manager.cc:1073] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
W0619 00:12:21.400391 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.405102 27199 ts_tablet_manager.cc:1382] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.405351 27199 log.cc:986] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/b80e7e9623ae4a229aa5df9a9c06b327
I0619 00:12:21.405939 27199 ts_tablet_manager.cc:1403] T b80e7e9623ae4a229aa5df9a9c06b327 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
W0619 00:12:21.408639 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.409744 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87f31dc0733b4059aa0d5d85999d1ea5" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:21.409961 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87f31dc0733b4059aa0d5d85999d1ea5" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:21.410217 26949 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.783s (vote granted)
I0619 00:12:21.410434 26949 raft_consensus.cc:2245] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:21.410439 26811 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.940s (vote granted)
I0619 00:12:21.410698 26811 raft_consensus.cc:2245] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:21.411361 27200 ts_tablet_manager.cc:1382] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.411460 27013 leader_election.cc:379] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.411731 27013 leader_election.cc:270] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:21.411964 27200 log.cc:986] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/b80e7e9623ae4a229aa5df9a9c06b327
I0619 00:12:21.412255 27435 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.826s (election complete)
I0619 00:12:21.412478 27435 raft_consensus.cc:2539] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:21.412675 27435 raft_consensus.cc:457] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:21.412744 27015 leader_election.cc:379] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.412936 27435 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.541s (starting election)
I0619 00:12:21.413152 27435 raft_consensus.cc:2784] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.415002 27196 ts_tablet_manager.cc:1073] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.000s
I0619 00:12:21.417443 27435 raft_consensus.cc:479] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.418375 27435 leader_election.cc:243] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.418488 27195 tablet_bootstrap.cc:427] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:21.418794 27435 leader_election.cc:243] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.418902 27195 ts_tablet_manager.cc:1045] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.051s	user 0.006s	sys 0.005s
I0619 00:12:21.420195 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87f31dc0733b4059aa0d5d85999d1ea5" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:21.420627 26949 raft_consensus.cc:2784] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.420894 26949 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.752s (vote granted)
I0619 00:12:21.421365 27195 raft_consensus.cc:325] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.422008 27195 raft_consensus.cc:351] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.422216 27195 raft_consensus.cc:628] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:21.419809 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "87f31dc0733b4059aa0d5d85999d1ea5" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:21.420197 27200 ts_tablet_manager.cc:1403] T b80e7e9623ae4a229aa5df9a9c06b327 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:21.423439 26811 raft_consensus.cc:2784] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.423713 26811 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.729s (vote granted)
I0619 00:12:21.425370 26949 raft_consensus.cc:2245] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:21.426710 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet b80e7e9623ae4a229aa5df9a9c06b327 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:21.428736 27083 raft_consensus.cc:1081] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:21.430299 27013 leader_election.cc:379] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.430083 27424 consensus_queue.cc:963] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:21.430524 27013 leader_election.cc:270] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:21.431908 26811 raft_consensus.cc:2245] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:21.432224 27472 raft_consensus.cc:2732] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.101s (election complete)
I0619 00:12:21.432512 27472 raft_consensus.cc:2539] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:21.422693 27195 consensus_queue.cc:227] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.433310 27015 leader_election.cc:379] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.433990 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet b80e7e9623ae4a229aa5df9a9c06b327 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:21.440716 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.442275 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.445760 27195 ts_tablet_manager.cc:1073] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.004s
W0619 00:12:21.446497 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.448510 27472 raft_consensus.cc:593] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:21.449033 27472 consensus_queue.cc:204] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.455832 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.458361 26949 raft_consensus.cc:1081] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:21.460397 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.460374 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 87f31dc0733b4059aa0d5d85999d1ea5 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.460763 27429 consensus_queue.cc:963] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:21.465330 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.470360 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 87f31dc0733b4059aa0d5d85999d1ea5
I0619 00:12:21.470674 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 87f31dc0733b4059aa0d5d85999d1ea5 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.471065 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 87f31dc0733b4059aa0d5d85999d1ea5 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.471428 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 87f31dc0733b4059aa0d5d85999d1ea5 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.471585 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 87f31dc0733b4059aa0d5d85999d1ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet c7036d5007eb4529b6935a47faf868e3) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.472079 27200 tablet_replica.cc:261] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.472311 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.472504 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.472703 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.472703 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 87f31dc0733b4059aa0d5d85999d1ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet c7036d5007eb4529b6935a47faf868e3) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.472889 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.473104 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.473145 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 87f31dc0733b4059aa0d5d85999d1ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet c7036d5007eb4529b6935a47faf868e3) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.473294 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.473470 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.473619 27199 tablet_replica.cc:261] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.473665 27200 raft_consensus.cc:2042] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.473871 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.473922 27200 raft_consensus.cc:2069] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.474063 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.474249 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.474429 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.474565 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 87f31dc0733b4059aa0d5d85999d1ea5 deleted and replaced by tablet c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.474629 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.474843 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.475029 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.475440 27200 ts_tablet_manager.cc:1369] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.475463 27210 tablet_replica.cc:261] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.475720 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.475919 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.476104 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.476290 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.476511 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.476727 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.476907 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(87f31dc0733b4059aa0d5d85999d1ea5)
I0619 00:12:21.477075 27210 raft_consensus.cc:2042] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.477329 27210 raft_consensus.cc:2069] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.478783 27210 ts_tablet_manager.cc:1369] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.482239 27199 raft_consensus.cc:2042] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:21.482651 27199 consensus_peers.cc:456] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:21.482892 27199 consensus_peers.cc:456] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:21.483245 27199 pending_rounds.cc:63] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:21.483397 27199 pending_rounds.cc:68] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391030581002240 op_type: NO_OP noop_request { }
I0619 00:12:21.483707 27199 raft_consensus.cc:2622] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:21.483932 27199 raft_consensus.cc:2069] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.485285 27062 tablet_service.cc:759] Processing CreateTablet for tablet c7036d5007eb4529b6935a47faf868e3 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.485910 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c7036d5007eb4529b6935a47faf868e3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.485821 26532 catalog_manager.cc:3663] Got report from deleted tablet 87f31dc0733b4059aa0d5d85999d1ea5 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet c7036d5007eb4529b6935a47faf868e3): Sending delete request for this tablet
I0619 00:12:21.486100 26792 tablet_service.cc:759] Processing CreateTablet for tablet c7036d5007eb4529b6935a47faf868e3 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.486788 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 87f31dc0733b4059aa0d5d85999d1ea5 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.487939 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 87f31dc0733b4059aa0d5d85999d1ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet c7036d5007eb4529b6935a47faf868e3) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.488160 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c7036d5007eb4529b6935a47faf868e3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.489089 26929 tablet_service.cc:759] Processing CreateTablet for tablet c7036d5007eb4529b6935a47faf868e3 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.489666 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c7036d5007eb4529b6935a47faf868e3. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:21.490285 27491 consensus_queue.cc:1042] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2c1dfdaa6d442539e4342d184d2167c" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I0619 00:12:21.497241 26792 ts_tablet_manager.cc:1173] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.504123 27196 ts_tablet_manager.cc:1028] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:21.504495 27196 tablet_bootstrap.cc:427] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:21.505525 27199 ts_tablet_manager.cc:1369] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.506769 27196 tablet_bootstrap.cc:578] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
W0619 00:12:21.511145 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.511598 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet c7036d5007eb4529b6935a47faf868e3 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.518985 27062 ts_tablet_manager.cc:1173] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.508635 26929 ts_tablet_manager.cc:1173] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.520129 27197 ts_tablet_manager.cc:1028] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:21.520402 27195 ts_tablet_manager.cc:1028] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:21.520483 27197 tablet_bootstrap.cc:427] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:21.520776 27195 tablet_bootstrap.cc:427] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:21.520828 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.521113 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.522151 26928 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.522711 27197 tablet_bootstrap.cc:578] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:21.522924 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.523031 27195 tablet_bootstrap.cc:578] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:21.523900 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.525981 26532 catalog_manager.cc:4596] ReplaceTablet: tablet c7036d5007eb4529b6935a47faf868e3 deleted and replaced by tablet 03b25458ac8b496bb61f747df97621f5
I0619 00:12:21.526976 26792 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.533776 27210 ts_tablet_manager.cc:1382] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.534051 27210 log.cc:986] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/87f31dc0733b4059aa0d5d85999d1ea5
I0619 00:12:21.534528 27210 ts_tablet_manager.cc:1403] T 87f31dc0733b4059aa0d5d85999d1ea5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.535965 27196 tablet_bootstrap.cc:427] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:21.536324 27196 ts_tablet_manager.cc:1045] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.032s	user 0.004s	sys 0.001s
I0619 00:12:21.536249 26929 tablet_service.cc:759] Processing CreateTablet for tablet 03b25458ac8b496bb61f747df97621f5 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.536909 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 03b25458ac8b496bb61f747df97621f5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.537114 26792 tablet_service.cc:759] Processing CreateTablet for tablet 03b25458ac8b496bb61f747df97621f5 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.537722 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 03b25458ac8b496bb61f747df97621f5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.540807 27062 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.537722 27196 raft_consensus.cc:325] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.541503 27063 tablet_service.cc:759] Processing CreateTablet for tablet 03b25458ac8b496bb61f747df97621f5 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.541839 27196 raft_consensus.cc:351] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.542021 27196 raft_consensus.cc:628] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:21.542112 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 03b25458ac8b496bb61f747df97621f5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.543084 27196 consensus_queue.cc:227] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.545718 27197 tablet_bootstrap.cc:427] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:21.546084 27197 ts_tablet_manager.cc:1045] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.026s	user 0.002s	sys 0.006s
I0619 00:12:21.547395 27197 raft_consensus.cc:325] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.548028 27197 raft_consensus.cc:351] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.548233 27197 raft_consensus.cc:628] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:21.548774 27197 consensus_queue.cc:227] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.550840 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.551981 26533 catalog_manager.cc:3663] Got report from deleted tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 03b25458ac8b496bb61f747df97621f5): Sending delete request for this tablet
I0619 00:12:21.552670 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.553370 27196 ts_tablet_manager.cc:1073] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I0619 00:12:21.554505 26791 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.1:39548
W0619 00:12:21.555168 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.555387 26533 catalog_manager.cc:3663] Got report from deleted tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 03b25458ac8b496bb61f747df97621f5): Sending delete request for this tablet
I0619 00:12:21.556141 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.546008 27200 ts_tablet_manager.cc:1382] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:12:21.557054 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.558514 26928 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.559777 27195 tablet_bootstrap.cc:427] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:21.560127 27195 ts_tablet_manager.cc:1045] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.039s	user 0.003s	sys 0.005s
I0619 00:12:21.560472 26532 catalog_manager.cc:3663] Got report from deleted tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 03b25458ac8b496bb61f747df97621f5): Sending delete request for this tablet
I0619 00:12:21.560585 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 87f31dc0733b4059aa0d5d85999d1ea5 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:21.561151 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
W0619 00:12:21.561398 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.562073 26928 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.563314 27197 ts_tablet_manager.cc:1073] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.001s
I0619 00:12:21.563370 27063 ts_tablet_manager.cc:1173] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.563035 27195 raft_consensus.cc:325] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.563693 27195 raft_consensus.cc:351] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.563905 27195 raft_consensus.cc:628] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:21.564631 27199 ts_tablet_manager.cc:1382] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.564918 27199 log.cc:986] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/87f31dc0733b4059aa0d5d85999d1ea5
I0619 00:12:21.565502 27199 ts_tablet_manager.cc:1403] T 87f31dc0733b4059aa0d5d85999d1ea5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:21.566083 27328 raft_consensus.cc:457] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:21.566325 27328 raft_consensus.cc:2732] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.985s (starting election)
I0619 00:12:21.566558 27328 raft_consensus.cc:479] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.567226 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 87f31dc0733b4059aa0d5d85999d1ea5 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:21.567534 27328 leader_election.cc:243] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:21.567813 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 87f31dc0733b4059aa0d5d85999d1ea5 because the tablet was not found. No further retry: Not found: Tablet not found: 87f31dc0733b4059aa0d5d85999d1ea5
I0619 00:12:21.567945 27328 leader_election.cc:243] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.568641 27210 tablet_replica.cc:261] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.568914 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.569106 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.569283 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.569468 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.569656 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.569505 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c7036d5007eb4529b6935a47faf868e3" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:21.569866 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.570046 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.570009 26811 raft_consensus.cc:2732] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.226s (vote granted)
I0619 00:12:21.569999 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c7036d5007eb4529b6935a47faf868e3" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:21.570212 27210 raft_consensus.cc:2042] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.570226 26811 raft_consensus.cc:2245] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:21.570443 27210 raft_consensus.cc:2069] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.564427 27195 consensus_queue.cc:227] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.570921 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c7036d5007eb4529b6935a47faf868e3 already in progress: creating tablet
I0619 00:12:21.571249 27083 raft_consensus.cc:2732] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.877s (vote granted)
I0619 00:12:21.571285 26876 leader_election.cc:379] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.571171 26521 catalog_manager.cc:2891] Scheduling retry of c7036d5007eb4529b6935a47faf868e3 Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 16 ms (attempt = 1)
I0619 00:12:21.571466 27083 raft_consensus.cc:2245] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:21.571535 26876 leader_election.cc:270] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:12:21.571861 26521 catalog_manager.cc:3083] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c7036d5007eb4529b6935a47faf868e3 already in progress: creating tablet
I0619 00:12:21.572067 26521 catalog_manager.cc:2891] Scheduling retry of c7036d5007eb4529b6935a47faf868e3 Delete Tablet RPC for TS=a2c1dfdaa6d442539e4342d184d2167c with a delay of 63 ms (attempt = 1)
I0619 00:12:21.572459 26874 leader_election.cc:379] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
W0619 00:12:21.573668 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.573722 26876 raft_consensus.cc:2442] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:12:21.574369 26532 catalog_manager.cc:3663] Got report from deleted tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 03b25458ac8b496bb61f747df97621f5): Sending delete request for this tablet
I0619 00:12:21.576020 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
W0619 00:12:21.576848 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.579869 26532 catalog_manager.cc:3663] Got report from deleted tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 03b25458ac8b496bb61f747df97621f5): Sending delete request for this tablet
I0619 00:12:21.580500 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.581097 26792 ts_tablet_manager.cc:1173] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.581497 27062 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.582026 27196 ts_tablet_manager.cc:1028] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:21.582363 27196 tablet_bootstrap.cc:427] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:21.583372 27328 raft_consensus.cc:457] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:21.583607 27328 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.827s (starting election)
I0619 00:12:21.583878 27328 raft_consensus.cc:479] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.584718 26929 ts_tablet_manager.cc:1173] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.585467 27196 tablet_bootstrap.cc:578] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:21.584882 27328 leader_election.cc:243] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.585939 27197 ts_tablet_manager.cc:1028] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:21.585927 27328 leader_election.cc:243] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.586283 27063 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.586467 27197 tablet_bootstrap.cc:427] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:21.586623 27200 log.cc:986] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/87f31dc0733b4059aa0d5d85999d1ea5
I0619 00:12:21.587894 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "59ec58633baa442bb7edbddc1655bd38" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:21.588348 26811 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.783s (vote granted)
I0619 00:12:21.588243 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "59ec58633baa442bb7edbddc1655bd38" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:21.588559 26811 raft_consensus.cc:2245] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:21.588762 27083 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.928s (vote granted)
I0619 00:12:21.588973 27083 raft_consensus.cc:2245] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:21.589459 27200 ts_tablet_manager.cc:1403] T 87f31dc0733b4059aa0d5d85999d1ea5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:21.589571 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.590150 26874 leader_election.cc:379] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.590395 26874 leader_election.cc:270] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:21.590539 26928 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.591222 26876 leader_election.cc:379] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.591317 27200 tablet_replica.cc:261] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.591477 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 87f31dc0733b4059aa0d5d85999d1ea5 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:21.591644 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.591671 27197 tablet_bootstrap.cc:578] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:21.591708 27328 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.883s (election complete)
I0619 00:12:21.591845 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.592020 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.591977 27328 raft_consensus.cc:2539] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:21.592201 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.592175 27328 raft_consensus.cc:457] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:21.592363 27328 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.905s (starting election)
I0619 00:12:21.592412 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.592551 27328 raft_consensus.cc:2784] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.592633 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(c7036d5007eb4529b6935a47faf868e3)
W0619 00:12:21.592620 26521 catalog_manager.cc:3083] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c7036d5007eb4529b6935a47faf868e3 already in progress: creating tablet
I0619 00:12:21.592813 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.592880 26521 catalog_manager.cc:2891] Scheduling retry of c7036d5007eb4529b6935a47faf868e3 Delete Tablet RPC for TS=a2c1dfdaa6d442539e4342d184d2167c with a delay of 24 ms (attempt = 1)
I0619 00:12:21.592981 27200 raft_consensus.cc:2042] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.593232 27200 raft_consensus.cc:2069] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:21.593263 26521 catalog_manager.cc:3083] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c7036d5007eb4529b6935a47faf868e3 already in progress: creating tablet
I0619 00:12:21.593458 26521 catalog_manager.cc:2891] Scheduling retry of c7036d5007eb4529b6935a47faf868e3 Delete Tablet RPC for TS=a2c1dfdaa6d442539e4342d184d2167c with a delay of 45 ms (attempt = 1)
I0619 00:12:21.580891 27195 ts_tablet_manager.cc:1073] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I0619 00:12:21.595772 27195 ts_tablet_manager.cc:1028] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:21.596133 27195 tablet_bootstrap.cc:427] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:21.596868 27328 raft_consensus.cc:479] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.597893 27328 leader_election.cc:243] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.598798 27328 leader_election.cc:243] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.599359 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "59ec58633baa442bb7edbddc1655bd38" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:21.599830 26811 raft_consensus.cc:2784] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.600073 26811 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.020s (vote granted)
I0619 00:12:21.600234 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "59ec58633baa442bb7edbddc1655bd38" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:21.600688 27083 raft_consensus.cc:2784] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.600916 27083 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.888s (vote granted)
I0619 00:12:21.604670 26811 raft_consensus.cc:2245] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:21.605712 26876 leader_election.cc:379] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.605805 27083 raft_consensus.cc:2245] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:21.605985 26876 leader_election.cc:270] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:21.606890 26874 leader_election.cc:379] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.607846 27210 ts_tablet_manager.cc:1369] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.608304 27197 tablet_bootstrap.cc:427] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:21.608621 27197 ts_tablet_manager.cc:1045] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.001s
I0619 00:12:21.610107 27197 raft_consensus.cc:325] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.610736 27195 tablet_bootstrap.cc:578] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:21.610812 27197 raft_consensus.cc:351] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.610997 27197 raft_consensus.cc:628] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:21.611493 27197 consensus_queue.cc:227] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.612130 27200 ts_tablet_manager.cc:1369] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.617751 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.618168 27196 tablet_bootstrap.cc:427] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:21.618496 27196 ts_tablet_manager.cc:1045] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.036s	user 0.005s	sys 0.001s
I0619 00:12:21.618732 27328 raft_consensus.cc:2732] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.107s (election complete)
I0619 00:12:21.618783 27063 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.619014 27328 raft_consensus.cc:2539] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:21.619287 27199 tablet_replica.cc:261] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.619525 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.619740 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.619920 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.620107 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.620321 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.619935 27196 raft_consensus.cc:325] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.620507 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.620532 27196 raft_consensus.cc:351] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.620715 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(c7036d5007eb4529b6935a47faf868e3)
I0619 00:12:21.620746 27196 raft_consensus.cc:628] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:21.620893 27199 raft_consensus.cc:2042] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.621148 27199 raft_consensus.cc:2069] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.621171 27196 consensus_queue.cc:227] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.622524 27199 ts_tablet_manager.cc:1369] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.623211 27328 raft_consensus.cc:593] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
W0619 00:12:21.624116 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.623690 27328 consensus_queue.cc:204] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.626905 27197 ts_tablet_manager.cc:1073] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.001s
W0619 00:12:21.629405 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.630939 27196 ts_tablet_manager.cc:1073] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
W0619 00:12:21.631420 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.631877 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.634075 26532 catalog_manager.cc:3786] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:21.636534 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.637604 27063 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.639547 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c7036d5007eb4529b6935a47faf868e3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 03b25458ac8b496bb61f747df97621f5)
I0619 00:12:21.640547 27063 tablet_service.cc:800] Processing DeleteTablet for tablet c7036d5007eb4529b6935a47faf868e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b25458ac8b496bb61f747df97621f5) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:21.641240 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.648686 27210 ts_tablet_manager.cc:1382] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.648938 27210 log.cc:986] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.649407 27210 ts_tablet_manager.cc:1403] T c7036d5007eb4529b6935a47faf868e3 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.651245 27200 ts_tablet_manager.cc:1382] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.651283 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:21.652130 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 because the tablet was not found. No further retry: Not found: Tablet not found: c7036d5007eb4529b6935a47faf868e3
W0619 00:12:21.652540 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 because the tablet was not found. No further retry: Not found: Tablet not found: c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.653009 27195 tablet_bootstrap.cc:427] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:21.653339 27195 ts_tablet_manager.cc:1045] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.057s	user 0.004s	sys 0.003s
I0619 00:12:21.654691 27195 raft_consensus.cc:325] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.655268 27195 raft_consensus.cc:351] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.655444 27195 raft_consensus.cc:628] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:21.655884 27195 consensus_queue.cc:227] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
W0619 00:12:21.658318 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.659652 27195 ts_tablet_manager.cc:1073] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
W0619 00:12:21.661114 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.666471 27200 log.cc:986] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.666982 27200 ts_tablet_manager.cc:1403] T c7036d5007eb4529b6935a47faf868e3 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:21.668689 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:21.655841 27199 ts_tablet_manager.cc:1382] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.673923 27199 log.cc:986] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.674415 27199 ts_tablet_manager.cc:1403] T c7036d5007eb4529b6935a47faf868e3 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
W0619 00:12:21.674566 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 because the tablet was not found. No further retry: Not found: Tablet not found: c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.676363 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet c7036d5007eb4529b6935a47faf868e3 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:21.677253 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 because the tablet was not found. No further retry: Not found: Tablet not found: c7036d5007eb4529b6935a47faf868e3
W0619 00:12:21.677702 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet c7036d5007eb4529b6935a47faf868e3 because the tablet was not found. No further retry: Not found: Tablet not found: c7036d5007eb4529b6935a47faf868e3
I0619 00:12:21.679265 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 59ec58633baa442bb7edbddc1655bd38 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.690502 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 59ec58633baa442bb7edbddc1655bd38
I0619 00:12:21.690810 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 59ec58633baa442bb7edbddc1655bd38 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet f4049071d2db4c629a9450fc85832e37)
I0619 00:12:21.691195 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 59ec58633baa442bb7edbddc1655bd38 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet f4049071d2db4c629a9450fc85832e37)
I0619 00:12:21.691565 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 59ec58633baa442bb7edbddc1655bd38 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet f4049071d2db4c629a9450fc85832e37)
I0619 00:12:21.691898 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 59ec58633baa442bb7edbddc1655bd38 deleted and replaced by tablet f4049071d2db4c629a9450fc85832e37
I0619 00:12:21.692399 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 59ec58633baa442bb7edbddc1655bd38 with delete_type TABLET_DATA_DELETED (replaced by tablet f4049071d2db4c629a9450fc85832e37) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.692901 27200 tablet_replica.cc:261] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.693145 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.693356 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.693548 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.693755 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.693964 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.694141 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.694308 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.694347 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 59ec58633baa442bb7edbddc1655bd38 with delete_type TABLET_DATA_DELETED (replaced by tablet f4049071d2db4c629a9450fc85832e37) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.694468 27200 raft_consensus.cc:2042] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.694748 27200 raft_consensus.cc:2069] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.695386 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 59ec58633baa442bb7edbddc1655bd38 with delete_type TABLET_DATA_DELETED (replaced by tablet f4049071d2db4c629a9450fc85832e37) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.695559 27210 tablet_replica.cc:261] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.695817 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.696022 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.696239 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.696436 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.696694 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.696898 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.697088 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.697269 27210 raft_consensus.cc:2042] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:21.697474 27210 consensus_peers.cc:456] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:21.697707 27210 consensus_peers.cc:456] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:21.698066 27210 pending_rounds.cc:63] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:21.698215 27210 pending_rounds.cc:68] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391031297114112 op_type: NO_OP noop_request { }
I0619 00:12:21.698925 27199 tablet_replica.cc:261] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.699167 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.699391 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.699584 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.699802 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.700024 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.700225 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.700417 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(59ec58633baa442bb7edbddc1655bd38)
I0619 00:12:21.700601 27199 raft_consensus.cc:2042] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.700765 27200 ts_tablet_manager.cc:1369] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.700898 27199 raft_consensus.cc:2069] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.702224 27199 ts_tablet_manager.cc:1369] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.698520 27210 raft_consensus.cc:2622] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:21.705801 27210 raft_consensus.cc:2069] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.709466 27063 tablet_service.cc:759] Processing CreateTablet for tablet f4049071d2db4c629a9450fc85832e37 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.710131 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f4049071d2db4c629a9450fc85832e37. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.713090 26929 tablet_service.cc:759] Processing CreateTablet for tablet f4049071d2db4c629a9450fc85832e37 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.713716 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f4049071d2db4c629a9450fc85832e37. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.718087 27210 ts_tablet_manager.cc:1369] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.727382 27210 ts_tablet_manager.cc:1382] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.727663 27210 log.cc:986] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/59ec58633baa442bb7edbddc1655bd38
I0619 00:12:21.729291 27210 ts_tablet_manager.cc:1403] T 59ec58633baa442bb7edbddc1655bd38 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.729955 27199 ts_tablet_manager.cc:1382] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.730759 27199 log.cc:986] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/59ec58633baa442bb7edbddc1655bd38
W0619 00:12:21.731123 26523 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918): Couldn't send request to peer 555bff44d4f942eaad13f790cca0a696 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times.
I0619 00:12:21.731343 27063 ts_tablet_manager.cc:1173] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.732630 26792 tablet_service.cc:759] Processing CreateTablet for tablet f4049071d2db4c629a9450fc85832e37 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.733192 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f4049071d2db4c629a9450fc85832e37. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.731279 27199 ts_tablet_manager.cc:1403] T 59ec58633baa442bb7edbddc1655bd38 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:21.733444 27195 ts_tablet_manager.cc:1028] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:21.733919 27195 tablet_bootstrap.cc:427] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:21.733011 26929 ts_tablet_manager.cc:1173] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.736223 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 59ec58633baa442bb7edbddc1655bd38 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:21.736636 27197 ts_tablet_manager.cc:1028] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:21.736990 27197 tablet_bootstrap.cc:427] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:21.739274 27197 tablet_bootstrap.cc:578] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:21.740309 27195 tablet_bootstrap.cc:578] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:21.742002 26792 ts_tablet_manager.cc:1173] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.742364 27196 ts_tablet_manager.cc:1028] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:21.742729 27196 tablet_bootstrap.cc:427] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:21.745512 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet e719602f08ff4deab45c4ae9a231fe94 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.747046 27196 tablet_bootstrap.cc:578] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:21.747995 27195 tablet_bootstrap.cc:427] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:21.750283 27195 ts_tablet_manager.cc:1045] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.001s
I0619 00:12:21.750743 26811 raft_consensus.cc:1081] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:21.751588 27195 raft_consensus.cc:325] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.758013 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet e719602f08ff4deab45c4ae9a231fe94
I0619 00:12:21.756021 27472 consensus_queue.cc:963] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:21.758298 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e719602f08ff4deab45c4ae9a231fe94 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 2881a156f46a461591799b815476c5e6)
I0619 00:12:21.758678 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e719602f08ff4deab45c4ae9a231fe94 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 2881a156f46a461591799b815476c5e6)
I0619 00:12:21.759021 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e719602f08ff4deab45c4ae9a231fe94 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 2881a156f46a461591799b815476c5e6)
I0619 00:12:21.759307 26532 catalog_manager.cc:4596] ReplaceTablet: tablet e719602f08ff4deab45c4ae9a231fe94 deleted and replaced by tablet 2881a156f46a461591799b815476c5e6
I0619 00:12:21.762205 27063 tablet_service.cc:800] Processing DeleteTablet for tablet e719602f08ff4deab45c4ae9a231fe94 with delete_type TABLET_DATA_DELETED (replaced by tablet 2881a156f46a461591799b815476c5e6) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.764971 27199 tablet_replica.cc:261] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.766438 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.766716 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.766930 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.767139 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(e719602f08ff4deab45c4ae9a231fe94)
W0619 00:12:21.767273 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.767390 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.767653 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.767874 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.768059 27199 raft_consensus.cc:2042] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:21.768268 27199 consensus_peers.cc:456] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:21.768496 27199 consensus_peers.cc:456] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:21.769716 27196 tablet_bootstrap.cc:427] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:21.770048 27196 ts_tablet_manager.cc:1045] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.027s	user 0.003s	sys 0.001s
I0619 00:12:21.771484 27196 raft_consensus.cc:325] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.772171 27196 raft_consensus.cc:351] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.772388 27196 raft_consensus.cc:628] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:21.772960 27196 consensus_queue.cc:227] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.774392 26928 tablet_service.cc:759] Processing CreateTablet for tablet 2881a156f46a461591799b815476c5e6 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.775017 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2881a156f46a461591799b815476c5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.778368 27063 tablet_service.cc:759] Processing CreateTablet for tablet 2881a156f46a461591799b815476c5e6 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.778992 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2881a156f46a461591799b815476c5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.780381 26791 tablet_service.cc:759] Processing CreateTablet for tablet 2881a156f46a461591799b815476c5e6 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.780982 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2881a156f46a461591799b815476c5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.783340 26532 catalog_manager.cc:3663] Got report from deleted tablet e719602f08ff4deab45c4ae9a231fe94 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) (replaced by tablet 2881a156f46a461591799b815476c5e6): Sending delete request for this tablet
I0619 00:12:21.783970 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e719602f08ff4deab45c4ae9a231fe94 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 2881a156f46a461591799b815476c5e6)
I0619 00:12:21.787008 26790 tablet_service.cc:800] Processing DeleteTablet for tablet e719602f08ff4deab45c4ae9a231fe94 with delete_type TABLET_DATA_DELETED (replaced by tablet 2881a156f46a461591799b815476c5e6) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.789788 27195 raft_consensus.cc:351] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.789813 26791 ts_tablet_manager.cc:1173] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.789999 27195 raft_consensus.cc:628] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:21.790205 26792 tablet_service.cc:800] Processing DeleteTablet for tablet e719602f08ff4deab45c4ae9a231fe94 with delete_type TABLET_DATA_DELETED (replaced by tablet 2881a156f46a461591799b815476c5e6) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.790549 27195 consensus_queue.cc:227] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.791723 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.792994 27195 ts_tablet_manager.cc:1073] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.042s	user 0.002s	sys 0.001s
W0619 00:12:21.793874 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.794405 26929 tablet_service.cc:800] Processing DeleteTablet for tablet e719602f08ff4deab45c4ae9a231fe94 with delete_type TABLET_DATA_DELETED (replaced by tablet 2881a156f46a461591799b815476c5e6) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.794327 27196 ts_tablet_manager.cc:1073] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.024s	user 0.003s	sys 0.001s
I0619 00:12:21.794920 27196 ts_tablet_manager.cc:1028] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:21.795265 27196 tablet_bootstrap.cc:427] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:21.796785 27063 ts_tablet_manager.cc:1173] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.798596 27196 tablet_bootstrap.cc:578] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:21.797276 27199 pending_rounds.cc:63] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
W0619 00:12:21.797919 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.799891 27199 pending_rounds.cc:68] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391029452894208 op_type: NO_OP noop_request { }
I0619 00:12:21.800444 27197 tablet_bootstrap.cc:427] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:21.800817 27197 ts_tablet_manager.cc:1045] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.064s	user 0.007s	sys 0.000s
W0619 00:12:21.800783 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.800971 27195 ts_tablet_manager.cc:1028] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:21.801489 27195 tablet_bootstrap.cc:427] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:21.802666 27197 raft_consensus.cc:325] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.801674 27199 raft_consensus.cc:2622] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:21.803268 27197 raft_consensus.cc:351] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.803369 27199 raft_consensus.cc:2069] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.803457 27197 raft_consensus.cc:628] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:21.803921 27197 consensus_queue.cc:227] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.806746 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.807075 27197 ts_tablet_manager.cc:1073] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
I0619 00:12:21.810153 26928 ts_tablet_manager.cc:1173] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:21.811413 27210 tablet_replica.cc:261] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.811514 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 59ec58633baa442bb7edbddc1655bd38 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:21.811686 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.811166 27199 ts_tablet_manager.cc:1369] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.811928 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.812108 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.812297 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.812512 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.812729 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.812918 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.813097 27210 raft_consensus.cc:2042] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.813356 27210 raft_consensus.cc:2069] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.814541 27210 ts_tablet_manager.cc:1369] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.816126 27195 tablet_bootstrap.cc:578] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:21.825582 27197 ts_tablet_manager.cc:1028] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:21.825987 27197 tablet_bootstrap.cc:427] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:21.828649 27197 tablet_bootstrap.cc:578] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:21.803052 27200 ts_tablet_manager.cc:1382] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.830600 27200 log.cc:986] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/59ec58633baa442bb7edbddc1655bd38
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0619 00:12:21.830540 27474 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0619 00:12:21.831145 27474 flags.cc:406] Enabled unsafe flag: --never_fsync=true
I0619 00:12:21.831198 27200 ts_tablet_manager.cc:1403] T 59ec58633baa442bb7edbddc1655bd38 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:21.832576 27196 tablet_bootstrap.cc:427] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:21.832955 27196 ts_tablet_manager.cc:1045] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.000s
I0619 00:12:21.833333 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 59ec58633baa442bb7edbddc1655bd38 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:21.834128 27200 tablet_replica.cc:261] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.834398 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.834641 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.834853 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.837906 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.838171 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.838402 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.839005 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(e719602f08ff4deab45c4ae9a231fe94)
I0619 00:12:21.839241 27200 raft_consensus.cc:2042] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.839326 27195 tablet_bootstrap.cc:427] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:21.839521 27200 pending_rounds.cc:63] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:21.839813 27195 ts_tablet_manager.cc:1045] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.038s	user 0.007s	sys 0.001s
I0619 00:12:21.839761 27200 pending_rounds.cc:68] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391029452894208 op_type: NO_OP noop_request { }
I0619 00:12:21.840128 27200 raft_consensus.cc:2622] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:21.840387 27200 raft_consensus.cc:2069] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.841492 27195 raft_consensus.cc:325] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.842120 27195 raft_consensus.cc:351] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.842306 27195 raft_consensus.cc:628] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:21.838940 27196 raft_consensus.cc:325] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.842757 27195 consensus_queue.cc:227] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.843204 27196 raft_consensus.cc:351] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.843387 27196 raft_consensus.cc:628] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:21.843873 27196 consensus_queue.cc:227] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.844986 27199 ts_tablet_manager.cc:1382] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W0619 00:12:21.847301 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.848685 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.848634 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.848984 27199 log.cc:986] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/e719602f08ff4deab45c4ae9a231fe94
I0619 00:12:21.849278 27197 tablet_bootstrap.cc:427] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:21.848744 27195 ts_tablet_manager.cc:1073] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I0619 00:12:21.849606 27197 ts_tablet_manager.cc:1045] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.024s	user 0.003s	sys 0.001s
I0619 00:12:21.850054 27199 ts_tablet_manager.cc:1403] T e719602f08ff4deab45c4ae9a231fe94 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:21.851361 27197 raft_consensus.cc:325] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.852447 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet e719602f08ff4deab45c4ae9a231fe94 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:21.852588 27197 raft_consensus.cc:351] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:21.852797 27197 raft_consensus.cc:628] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:21.853309 27197 consensus_queue.cc:227] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.853955 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.856786 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:21.858534 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.859817 27197 ts_tablet_manager.cc:1073] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I0619 00:12:21.860437 27196 ts_tablet_manager.cc:1073] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.001s
I0619 00:12:21.861943 27210 ts_tablet_manager.cc:1382] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.862210 27210 log.cc:986] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/e719602f08ff4deab45c4ae9a231fe94
I0619 00:12:21.862720 27210 ts_tablet_manager.cc:1403] T e719602f08ff4deab45c4ae9a231fe94 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.864508 27200 ts_tablet_manager.cc:1369] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.864979 27429 raft_consensus.cc:457] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:21.869069 27429 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.966s (starting election)
I0619 00:12:21.869319 27429 raft_consensus.cc:479] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.870306 27429 leader_election.cc:243] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.870695 27429 leader_election.cc:243] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.872406 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03b25458ac8b496bb61f747df97621f5" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:21.872920 27083 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.692s (vote granted)
I0619 00:12:21.873147 27083 raft_consensus.cc:2245] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:21.874150 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03b25458ac8b496bb61f747df97621f5" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:21.874624 26949 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.876s (vote granted)
I0619 00:12:21.874864 26949 raft_consensus.cc:2245] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:21.875396 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet e719602f08ff4deab45c4ae9a231fe94 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:21.875851 26732 leader_election.cc:379] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.876070 26732 leader_election.cc:270] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:21.876813 26732 leader_election.cc:379] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.877245 27429 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.512s (election complete)
I0619 00:12:21.877477 27429 raft_consensus.cc:2539] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:21.877691 27429 raft_consensus.cc:457] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:21.877880 27429 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.619s (starting election)
I0619 00:12:21.878073 27429 raft_consensus.cc:2784] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:12:21.878922 27474 flags.cc:406] Enabled experimental flag: --hive_metastore_sasl_enabled=false
W0619 00:12:21.879092 27474 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:55188
W0619 00:12:21.879252 27474 flags.cc:406] Enabled experimental flag: --hive_metastore_notification_log_poll_period_seconds=1
W0619 00:12:21.879397 27474 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
I0619 00:12:21.879333 27472 raft_consensus.cc:457] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W0619 00:12:21.879570 27474 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
I0619 00:12:21.879580 27472 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.522s (starting election)
W0619 00:12:21.879750 27474 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0619 00:12:21.879930 27474 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0619 00:12:21.879853 27472 raft_consensus.cc:479] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.880815 27472 leader_election.cc:243] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.881206 27472 leader_election.cc:243] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.882477 27429 raft_consensus.cc:479] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.882951 27499 raft_consensus.cc:457] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:21.883038 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2881a156f46a461591799b815476c5e6" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:21.883052 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2881a156f46a461591799b815476c5e6" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:21.883294 27499 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.596s (starting election)
I0619 00:12:21.883491 26811 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.896s (vote granted)
I0619 00:12:21.883555 26949 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.934s (vote granted)
I0619 00:12:21.883786 26811 raft_consensus.cc:2245] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:21.883805 26949 raft_consensus.cc:2245] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:21.883589 27499 raft_consensus.cc:479] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.884603 27499 leader_election.cc:243] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.884954 27015 leader_election.cc:379] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.885037 27499 leader_election.cc:243] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.885179 27015 leader_election.cc:270] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:21.885649 27472 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.937s (election complete)
I0619 00:12:21.883406 27429 leader_election.cc:243] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.885896 27472 raft_consensus.cc:2539] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:21.886096 27472 raft_consensus.cc:457] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:21.886287 27472 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.246s (starting election)
I0619 00:12:21.886484 27472 raft_consensus.cc:2784] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.887090 27429 leader_election.cc:243] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.887136 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f4049071d2db4c629a9450fc85832e37" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:21.887580 26811 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.872s (vote granted)
I0619 00:12:21.887568 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03b25458ac8b496bb61f747df97621f5" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:21.887826 26811 raft_consensus.cc:2245] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:21.887972 26949 raft_consensus.cc:2784] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.888190 26949 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.749s (vote granted)
I0619 00:12:21.888556 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "03b25458ac8b496bb61f747df97621f5" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:21.888998 27083 raft_consensus.cc:2784] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.889217 27083 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.657s (vote granted)
I0619 00:12:21.892045 27013 leader_election.cc:379] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.893146 27081 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f4049071d2db4c629a9450fc85832e37" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:21.893579 27081 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.568s (vote granted)
I0619 00:12:21.893818 27081 raft_consensus.cc:2245] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:21.894956 26874 leader_election.cc:379] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.895267 26874 leader_election.cc:270] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:21.895978 26949 raft_consensus.cc:2245] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:21.896647 27499 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.150s (election complete)
I0619 00:12:21.896916 26876 leader_election.cc:379] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.897876 26732 leader_election.cc:379] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.898111 26732 leader_election.cc:270] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:21.898633 27429 raft_consensus.cc:2732] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.103s (election complete)
I0619 00:12:21.898860 27429 raft_consensus.cc:2539] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:21.899157 27429 raft_consensus.cc:593] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:21.899245 27472 raft_consensus.cc:479] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.899559 27429 consensus_queue.cc:204] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:21.900188 27472 leader_election.cc:243] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
W0619 00:12:21.902788 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.903250 27083 raft_consensus.cc:2245] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:21.896916 27499 raft_consensus.cc:2539] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:21.904023 27499 raft_consensus.cc:457] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:21.904217 27499 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.610s (starting election)
I0619 00:12:21.904412 27499 raft_consensus.cc:2784] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.904621 26732 leader_election.cc:379] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.907963 27200 ts_tablet_manager.cc:1382] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:21.908215 27200 log.cc:986] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/e719602f08ff4deab45c4ae9a231fe94
I0619 00:12:21.908808 27200 ts_tablet_manager.cc:1403] T e719602f08ff4deab45c4ae9a231fe94 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:21.902398 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2881a156f46a461591799b815476c5e6" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:21.909077 26949 raft_consensus.cc:2784] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.913621 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet e719602f08ff4deab45c4ae9a231fe94 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:21.909669 27499 raft_consensus.cc:479] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.913983 26949 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.177s (vote granted)
I0619 00:12:21.914549 27499 leader_election.cc:243] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.915459 27472 leader_election.cc:243] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.916014 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f4049071d2db4c629a9450fc85832e37" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:21.916414 26811 raft_consensus.cc:2784] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.916636 26811 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.557s (vote granted)
I0619 00:12:21.916879 27499 leader_election.cc:243] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.916999 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2881a156f46a461591799b815476c5e6" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:21.917415 26812 raft_consensus.cc:2784] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.917657 26812 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.611s (vote granted)
I0619 00:12:21.918277 26949 raft_consensus.cc:2245] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:21.918407 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f4049071d2db4c629a9450fc85832e37" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:21.918860 27083 raft_consensus.cc:2784] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:21.919090 27083 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.995s (vote granted)
I0619 00:12:21.919751 27013 leader_election.cc:379] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:21.920007 27013 leader_election.cc:270] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:21.920433 27489 raft_consensus.cc:2732] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.830s (election complete)
I0619 00:12:21.920747 27489 raft_consensus.cc:2539] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:21.920781 26811 raft_consensus.cc:2245] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:21.921039 27489 raft_consensus.cc:593] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:21.921527 27489 consensus_queue.cc:204] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:21.922086 26812 raft_consensus.cc:2245] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:21.922538 26876 leader_election.cc:379] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.922797 26876 leader_election.cc:270] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
W0619 00:12:21.923419 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.923638 27328 raft_consensus.cc:2732] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.548s (election complete)
I0619 00:12:21.923673 27015 leader_election.cc:379] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:21.923866 27328 raft_consensus.cc:2539] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:21.924144 27328 raft_consensus.cc:593] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:21.924546 27328 consensus_queue.cc:204] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:21.928784 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.928306 26533 catalog_manager.cc:3786] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } }
I0619 00:12:21.934278 27083 raft_consensus.cc:2245] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:21.936216 26874 leader_election.cc:379] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:21.937731 26532 catalog_manager.cc:3786] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
W0619 00:12:21.942459 26523 catalog_manager.cc:3071] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): delete failed for tablet e719602f08ff4deab45c4ae9a231fe94 because the tablet was not found. No further retry: Not found: Tablet not found: e719602f08ff4deab45c4ae9a231fe94
I0619 00:12:21.944555 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 2881a156f46a461591799b815476c5e6 from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:21.957319 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:21.963757 26532 catalog_manager.cc:3786] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:21.964848 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 2881a156f46a461591799b815476c5e6
I0619 00:12:21.965142 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2881a156f46a461591799b815476c5e6 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:21.965528 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2881a156f46a461591799b815476c5e6 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:21.965922 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2881a156f46a461591799b815476c5e6 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:21.966126 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 2881a156f46a461591799b815476c5e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fac1a493e4a245b2b5a017e61f877e4f) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:21.966236 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 2881a156f46a461591799b815476c5e6 deleted and replaced by tablet fac1a493e4a245b2b5a017e61f877e4f
I0619 00:12:21.966593 27210 tablet_replica.cc:261] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:21.966855 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.967048 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.967231 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.967414 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.967655 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.967846 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.968035 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.968216 27210 raft_consensus.cc:2042] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.968258 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 2881a156f46a461591799b815476c5e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fac1a493e4a245b2b5a017e61f877e4f) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:21.968483 27210 raft_consensus.cc:2069] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.968768 27200 tablet_replica.cc:261] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:21.968999 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.969187 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.969355 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.969523 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.969499 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 2881a156f46a461591799b815476c5e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fac1a493e4a245b2b5a017e61f877e4f) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:21.969764 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.969944 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.969970 27199 tablet_replica.cc:261] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:21.970098 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.970075 27210 ts_tablet_manager.cc:1369] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.970196 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.970250 27200 raft_consensus.cc:2042] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:21.970384 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.970487 27200 raft_consensus.cc:2069] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.970561 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.970767 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.970975 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.971161 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.971339 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(2881a156f46a461591799b815476c5e6)
I0619 00:12:21.971508 27199 raft_consensus.cc:2042] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:21.971726 27199 consensus_peers.cc:456] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:21.971930 27199 consensus_peers.cc:456] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:21.972276 27199 pending_rounds.cc:63] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:21.972426 27199 pending_rounds.cc:68] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391032516812800 op_type: NO_OP noop_request { }
I0619 00:12:21.972734 27199 raft_consensus.cc:2622] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:21.972954 27199 raft_consensus.cc:2069] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:21.974530 27200 ts_tablet_manager.cc:1369] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.977288 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 2881a156f46a461591799b815476c5e6 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:21.988273 27199 ts_tablet_manager.cc:1369] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:21.989976 27210 ts_tablet_manager.cc:1382] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:21.990203 27210 log.cc:986] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/2881a156f46a461591799b815476c5e6
I0619 00:12:21.990737 27210 ts_tablet_manager.cc:1403] T 2881a156f46a461591799b815476c5e6 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:21.994011 27063 tablet_service.cc:759] Processing CreateTablet for tablet fac1a493e4a245b2b5a017e61f877e4f (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.994065 26928 tablet_service.cc:759] Processing CreateTablet for tablet fac1a493e4a245b2b5a017e61f877e4f (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.994642 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fac1a493e4a245b2b5a017e61f877e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.994482 26792 tablet_service.cc:759] Processing CreateTablet for tablet fac1a493e4a245b2b5a017e61f877e4f (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:21.994619 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fac1a493e4a245b2b5a017e61f877e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:21.995102 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fac1a493e4a245b2b5a017e61f877e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.004659 26928 ts_tablet_manager.cc:1173] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.004979 27063 ts_tablet_manager.cc:1173] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.005023 27197 ts_tablet_manager.cc:1028] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:22.005352 27197 tablet_bootstrap.cc:427] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:22.007588 27195 ts_tablet_manager.cc:1028] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:22.007601 27197 tablet_bootstrap.cc:578] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:22.007954 27195 tablet_bootstrap.cc:427] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:22.008280 26792 ts_tablet_manager.cc:1173] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.008693 27196 ts_tablet_manager.cc:1028] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:22.009052 27196 tablet_bootstrap.cc:427] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:22.010084 27195 tablet_bootstrap.cc:578] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:22.019968 27196 tablet_bootstrap.cc:578] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:22.020514 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 2881a156f46a461591799b815476c5e6 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:22.021442 27195 tablet_bootstrap.cc:427] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:22.021801 27195 ts_tablet_manager.cc:1045] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.002s
I0619 00:12:22.023313 27197 tablet_bootstrap.cc:427] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:22.023675 27197 ts_tablet_manager.cc:1045] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.002s
I0619 00:12:22.023130 27195 raft_consensus.cc:325] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.025419 27200 ts_tablet_manager.cc:1382] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.025542 27195 raft_consensus.cc:351] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.025096 27197 raft_consensus.cc:325] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.025707 27200 log.cc:986] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/2881a156f46a461591799b815476c5e6
I0619 00:12:22.025813 27197 raft_consensus.cc:351] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.026001 27197 raft_consensus.cc:628] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:22.026175 27200 ts_tablet_manager.cc:1403] T 2881a156f46a461591799b815476c5e6 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:22.025894 27195 raft_consensus.cc:628] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:22.026516 27197 consensus_queue.cc:227] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.030467 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.031092 27197 ts_tablet_manager.cc:1073] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I0619 00:12:22.027091 27195 consensus_queue.cc:227] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.033126 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 2881a156f46a461591799b815476c5e6 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
W0619 00:12:22.034440 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.035393 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.035773 27195 ts_tablet_manager.cc:1073] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
W0619 00:12:22.037077 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.037358 27196 tablet_bootstrap.cc:427] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:22.037698 27196 ts_tablet_manager.cc:1045] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.002s
I0619 00:12:22.025096 27199 ts_tablet_manager.cc:1382] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.038790 27199 log.cc:986] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/2881a156f46a461591799b815476c5e6
I0619 00:12:22.040345 27196 raft_consensus.cc:325] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.040959 27196 raft_consensus.cc:351] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.041137 27196 raft_consensus.cc:628] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:22.041519 27199 ts_tablet_manager.cc:1403] T 2881a156f46a461591799b815476c5e6 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:22.043443 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 2881a156f46a461591799b815476c5e6 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:22.042549 27196 consensus_queue.cc:227] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.045711 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.055007 27196 ts_tablet_manager.cc:1073] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.000s
W0619 00:12:22.055896 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.080777 27474 master_main.cc:73] Master server non-default flags:
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:55188
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=1024
--master_addresses=127.24.119.62:58195,127.24.119.61:43000,127.24.119.60:33918
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.119.60:33918
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.119.60
--webserver_port=32921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.8.0-SNAPSHOT
revision 4ea99dd079cd5d95c3e7f2e8653ae26e4896a3b6
build type FASTDEBUG
built by jenkins at 18 Jun 2018 22:56:43 PST on quasar-xmwjxn-1.vpc.cloudera.com
build id 80332
TSAN enabled
I0619 00:12:22.103794 27474 master_main.cc:80] Initializing master server...
I0619 00:12:22.107019 27474 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 459413us
I0619 00:12:22.108140 27474 fs_manager.cc:261] Metadata directory not provided
I0619 00:12:22.108348 27474 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0619 00:12:22.128095 27474 fs_manager.cc:397] Time spent opening directory manager: real 0.016s	user 0.003s	sys 0.009s
I0619 00:12:22.129391 27474 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0619 00:12:22.130872 27474 file_cache.cc:470] Constructed file cache lbm with capacity 13107
I0619 00:12:22.149238 27474 fs_manager.cc:417] Time spent opening block manager: real 0.007s	user 0.002s	sys 0.004s
I0619 00:12:22.149454 27474 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
uuid: "555bff44d4f942eaad13f790cca0a696"
format_stamp: "Formatted at 2018-06-19 07:12:12 on quasar-xmwjxn-1.vpc.cloudera.com"
I0619 00:12:22.151446 27474 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0619 00:12:22.167163 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:22.171185 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 03b25458ac8b496bb61f747df97621f5 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.180279 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 03b25458ac8b496bb61f747df97621f5
I0619 00:12:22.180568 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03b25458ac8b496bb61f747df97621f5 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.180984 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03b25458ac8b496bb61f747df97621f5 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.181635 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 03b25458ac8b496bb61f747df97621f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e1883f069634e849dcba5e44201993e) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:22.182021 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 03b25458ac8b496bb61f747df97621f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e1883f069634e849dcba5e44201993e) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:22.182148 27210 tablet_replica.cc:261] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:22.182373 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.182488 27199 tablet_replica.cc:261] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:22.182565 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.182735 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.182767 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.182931 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.182941 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183115 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183154 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183296 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183336 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183509 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183511 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183717 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183717 27210 raft_consensus.cc:2042] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.183751 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet f4049071d2db4c629a9450fc85832e37 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.183899 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.183980 27210 raft_consensus.cc:2069] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.184077 27199 raft_consensus.cc:2042] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.184327 27199 raft_consensus.cc:2069] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.185158 27210 ts_tablet_manager.cc:1369] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.181360 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 03b25458ac8b496bb61f747df97621f5 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.188135 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 03b25458ac8b496bb61f747df97621f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e1883f069634e849dcba5e44201993e) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:22.188514 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 03b25458ac8b496bb61f747df97621f5 deleted and replaced by tablet 1e1883f069634e849dcba5e44201993e
I0619 00:12:22.188599 27200 tablet_replica.cc:261] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:22.188859 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.189065 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.189244 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.189424 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.189666 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.189867 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.190057 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(03b25458ac8b496bb61f747df97621f5)
I0619 00:12:22.190238 27200 raft_consensus.cc:2042] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:22.190454 27200 consensus_peers.cc:456] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:22.190655 27200 consensus_peers.cc:456] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:22.191025 27200 pending_rounds.cc:63] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:22.191177 27200 pending_rounds.cc:68] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391032432316416 op_type: NO_OP noop_request { }
I0619 00:12:22.191475 27200 raft_consensus.cc:2622] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:22.185509 27199 ts_tablet_manager.cc:1369] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.192772 27200 raft_consensus.cc:2069] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.194315 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet f4049071d2db4c629a9450fc85832e37
I0619 00:12:22.194620 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f4049071d2db4c629a9450fc85832e37 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 272d0c23372d485595f52ce73d28f56c)
I0619 00:12:22.194999 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f4049071d2db4c629a9450fc85832e37 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 272d0c23372d485595f52ce73d28f56c)
I0619 00:12:22.195374 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet f4049071d2db4c629a9450fc85832e37 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 272d0c23372d485595f52ce73d28f56c)
I0619 00:12:22.195551 26928 tablet_service.cc:800] Processing DeleteTablet for tablet f4049071d2db4c629a9450fc85832e37 with delete_type TABLET_DATA_DELETED (replaced by tablet 272d0c23372d485595f52ce73d28f56c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:22.195708 26533 catalog_manager.cc:4596] ReplaceTablet: tablet f4049071d2db4c629a9450fc85832e37 deleted and replaced by tablet 272d0c23372d485595f52ce73d28f56c
I0619 00:12:22.196388 27210 ts_tablet_manager.cc:1382] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.196399 27063 tablet_service.cc:800] Processing DeleteTablet for tablet f4049071d2db4c629a9450fc85832e37 with delete_type TABLET_DATA_DELETED (replaced by tablet 272d0c23372d485595f52ce73d28f56c) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:22.196960 26792 tablet_service.cc:800] Processing DeleteTablet for tablet f4049071d2db4c629a9450fc85832e37 with delete_type TABLET_DATA_DELETED (replaced by tablet 272d0c23372d485595f52ce73d28f56c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:22.197093 27210 log.cc:986] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/03b25458ac8b496bb61f747df97621f5
I0619 00:12:22.198256 27210 ts_tablet_manager.cc:1403] T 03b25458ac8b496bb61f747df97621f5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:22.200320 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 03b25458ac8b496bb61f747df97621f5 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.201529 27200 ts_tablet_manager.cc:1369] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:22.206157 26523 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918): Couldn't send request to peer 555bff44d4f942eaad13f790cca0a696 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times.
I0619 00:12:22.212221 26928 tablet_service.cc:759] Processing CreateTablet for tablet 1e1883f069634e849dcba5e44201993e (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.212656 27063 tablet_service.cc:759] Processing CreateTablet for tablet 1e1883f069634e849dcba5e44201993e (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.212829 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1e1883f069634e849dcba5e44201993e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.213239 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1e1883f069634e849dcba5e44201993e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.214514 26792 tablet_service.cc:759] Processing CreateTablet for tablet 1e1883f069634e849dcba5e44201993e (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.215137 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1e1883f069634e849dcba5e44201993e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.217260 27210 tablet_replica.cc:261] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:22.217520 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.217741 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.217922 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.218106 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.218323 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.218523 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.218736 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.218912 27210 raft_consensus.cc:2042] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:22.219108 27210 consensus_peers.cc:456] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:22.219300 27210 consensus_peers.cc:456] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:22.219650 27210 pending_rounds.cc:63] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:22.219790 27210 pending_rounds.cc:68] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391032530259968 op_type: NO_OP noop_request { }
I0619 00:12:22.220072 27210 raft_consensus.cc:2622] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:22.220280 27210 raft_consensus.cc:2069] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.228521 27200 ts_tablet_manager.cc:1382] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.236338 26929 tablet_service.cc:759] Processing CreateTablet for tablet 272d0c23372d485595f52ce73d28f56c (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.236460 27063 ts_tablet_manager.cc:1173] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.236733 26791 tablet_service.cc:759] Processing CreateTablet for tablet 272d0c23372d485595f52ce73d28f56c (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.236976 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 272d0c23372d485595f52ce73d28f56c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.237318 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 272d0c23372d485595f52ce73d28f56c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.240401 27062 tablet_service.cc:759] Processing CreateTablet for tablet 272d0c23372d485595f52ce73d28f56c (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.241009 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 272d0c23372d485595f52ce73d28f56c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.241142 26792 ts_tablet_manager.cc:1173] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.241539 27196 ts_tablet_manager.cc:1028] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:22.241889 27196 tablet_bootstrap.cc:427] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:22.243890 27196 tablet_bootstrap.cc:578] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:22.245720 27199 ts_tablet_manager.cc:1382] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.245890 26791 ts_tablet_manager.cc:1173] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.245960 27199 log.cc:986] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/03b25458ac8b496bb61f747df97621f5
I0619 00:12:22.246419 27199 ts_tablet_manager.cc:1403] T 03b25458ac8b496bb61f747df97621f5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:22.247445 27210 ts_tablet_manager.cc:1369] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.249402 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 03b25458ac8b496bb61f747df97621f5 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.254694 27196 tablet_bootstrap.cc:427] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:22.255031 27196 ts_tablet_manager.cc:1045] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.002s
I0619 00:12:22.255853 27199 tablet_replica.cc:261] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:22.256145 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.256336 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.256525 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.256736 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.256955 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.256566 27196 raft_consensus.cc:325] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.257220 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.257405 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.257375 27196 raft_consensus.cc:351] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.257603 27199 raft_consensus.cc:2042] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.257612 27196 raft_consensus.cc:628] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:22.257867 27199 raft_consensus.cc:2069] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.258261 27196 consensus_queue.cc:227] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.259060 27199 ts_tablet_manager.cc:1369] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.228794 27200 log.cc:986] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/03b25458ac8b496bb61f747df97621f5
I0619 00:12:22.261195 27200 ts_tablet_manager.cc:1403] T 03b25458ac8b496bb61f747df97621f5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:22.261495 26928 ts_tablet_manager.cc:1173] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.261960 27197 ts_tablet_manager.cc:1028] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:22.262315 27197 tablet_bootstrap.cc:427] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:22.263216 27200 tablet_replica.cc:261] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:22.263454 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.263650 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.262614 27195 ts_tablet_manager.cc:1028] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:22.263859 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.264032 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.264664 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 03b25458ac8b496bb61f747df97621f5 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.264235 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.265231 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.265417 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(f4049071d2db4c629a9450fc85832e37)
I0619 00:12:22.265592 27200 raft_consensus.cc:2042] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
W0619 00:12:22.265871 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.266899 27197 tablet_bootstrap.cc:578] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:22.266953 26929 ts_tablet_manager.cc:1173] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.270910 27195 tablet_bootstrap.cc:427] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:22.270934 27062 ts_tablet_manager.cc:1173] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.265871 27200 raft_consensus.cc:2069] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.272672 27200 ts_tablet_manager.cc:1369] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.273033 27195 tablet_bootstrap.cc:578] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:22.276782 27328 raft_consensus.cc:457] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:22.277005 27328 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.600s (starting election)
I0619 00:12:22.277258 27328 raft_consensus.cc:479] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.278223 27328 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.278578 27328 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.279824 27472 raft_consensus.cc:457] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:22.280066 27472 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.683s (starting election)
I0619 00:12:22.280061 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:22.280143 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:22.280539 26811 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.938s (vote granted)
I0619 00:12:22.280777 26811 raft_consensus.cc:2245] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:22.280520 27472 raft_consensus.cc:479] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.281476 27472 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.281533 27083 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.956s (vote granted)
I0619 00:12:22.281810 27083 raft_consensus.cc:2245] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:22.281884 27472 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.282037 26876 leader_election.cc:379] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.282254 26876 leader_election.cc:270] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:22.282819 27328 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.905s (election complete)
I0619 00:12:22.283032 27328 raft_consensus.cc:2539] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:22.283218 27328 raft_consensus.cc:457] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:22.283418 27328 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.803s (starting election)
I0619 00:12:22.283332 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:22.283628 27328 raft_consensus.cc:2784] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.284260 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:22.284693 26811 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.979s (vote granted)
I0619 00:12:22.284898 26811 raft_consensus.cc:2245] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:22.286164 27197 tablet_bootstrap.cc:427] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:22.286480 27197 ts_tablet_manager.cc:1045] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.024s	user 0.002s	sys 0.005s
I0619 00:12:22.286456 27015 leader_election.cc:379] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.286706 27015 leader_election.cc:270] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:22.287046 26874 leader_election.cc:379] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.287184 27472 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.685s (election complete)
I0619 00:12:22.287405 27472 raft_consensus.cc:2539] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:22.287623 27472 raft_consensus.cc:457] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:22.287813 27472 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.891s (starting election)
I0619 00:12:22.288004 27472 raft_consensus.cc:2784] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.288518 27328 raft_consensus.cc:479] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.288991 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.289474 27328 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.289654 26949 raft_consensus.cc:2175] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:22.289865 27328 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.290840 27197 raft_consensus.cc:325] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.291334 27013 leader_election.cc:393] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Invalid argument: T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:22.291497 27197 raft_consensus.cc:351] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.291790 27197 raft_consensus.cc:628] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:22.291895 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:22.292325 26811 raft_consensus.cc:2784] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.292263 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:22.292562 26811 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.809s (vote granted)
I0619 00:12:22.292407 27197 consensus_queue.cc:227] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.293503 27472 raft_consensus.cc:479] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.295861 27083 raft_consensus.cc:2175] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 in current term 1: Already voted for candidate a2c1dfdaa6d442539e4342d184d2167c in this term.
I0619 00:12:22.297871 27472 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.298812 26874 leader_election.cc:393] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote denied by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806). Message: Invalid argument: T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f719e8be9c9447a9acd40728c0a5710 in current term 1: Already voted for candidate a2c1dfdaa6d442539e4342d184d2167c in this term.
W0619 00:12:22.298929 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.298326 27472 leader_election.cc:243] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.301038 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:22.299607 27197 ts_tablet_manager.cc:1073] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.001s
I0619 00:12:22.302462 27197 ts_tablet_manager.cc:1028] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
W0619 00:12:22.302938 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.302875 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fac1a493e4a245b2b5a017e61f877e4f" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:22.303424 26949 raft_consensus.cc:2175] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:22.303712 27513 raft_consensus.cc:457] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:22.303941 27197 tablet_bootstrap.cc:427] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:22.303980 27513 raft_consensus.cc:2732] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.057s (starting election)
I0619 00:12:22.304247 27513 raft_consensus.cc:479] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.305264 27513 leader_election.cc:243] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.305702 27513 leader_election.cc:243] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.306334 27197 tablet_bootstrap.cc:578] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:22.307257 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1e1883f069634e849dcba5e44201993e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
W0619 00:12:22.309542 26732 leader_election.cc:306] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:22.311121 27013 leader_election.cc:393] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Invalid argument: T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 1f719e8be9c9447a9acd40728c0a5710 in this term.
I0619 00:12:22.311995 27196 ts_tablet_manager.cc:1073] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.057s	user 0.004s	sys 0.000s
I0619 00:12:22.312835 27196 ts_tablet_manager.cc:1028] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:22.318058 27199 ts_tablet_manager.cc:1382] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.318305 27199 log.cc:986] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/f4049071d2db4c629a9450fc85832e37
I0619 00:12:22.312363 26811 raft_consensus.cc:2245] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:22.318754 26812 raft_consensus.cc:2220] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:12:22.307535 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1e1883f069634e849dcba5e44201993e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:22.319819 26949 raft_consensus.cc:2732] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.921s (vote granted)
I0619 00:12:22.320045 26949 raft_consensus.cc:2245] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:22.320137 26876 leader_election.cc:379] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.320402 26876 leader_election.cc:270] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:22.321020 27328 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.569s (election complete)
I0619 00:12:22.320358 27015 leader_election.cc:393] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote denied by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295). Message: Service unavailable: T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:12:22.321326 27328 raft_consensus.cc:2539] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:22.321492 27015 leader_election.cc:270] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I0619 00:12:22.321208 27195 tablet_bootstrap.cc:427] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:22.322060 26732 leader_election.cc:379] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.322067 27328 raft_consensus.cc:593] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:22.322294 26732 leader_election.cc:270] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:22.322468 27195 ts_tablet_manager.cc:1045] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.052s	user 0.005s	sys 0.004s
I0619 00:12:22.322986 27513 raft_consensus.cc:2732] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.965s (election complete)
I0619 00:12:22.323222 27513 raft_consensus.cc:2539] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:22.314941 27210 ts_tablet_manager.cc:1382] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.322926 27328 consensus_queue.cc:204] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.323456 27513 raft_consensus.cc:457] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:22.323504 27210 log.cc:986] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/f4049071d2db4c629a9450fc85832e37
I0619 00:12:22.323683 27513 raft_consensus.cc:2732] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.841s (starting election)
I0619 00:12:22.323880 27513 raft_consensus.cc:2784] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.324116 27210 ts_tablet_manager.cc:1403] T f4049071d2db4c629a9450fc85832e37 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:22.324334 27195 raft_consensus.cc:325] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.324944 27195 raft_consensus.cc:351] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.325121 27195 raft_consensus.cc:628] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:22.326102 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet f4049071d2db4c629a9450fc85832e37 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:22.328171 27472 raft_consensus.cc:2732] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.522s (election complete)
I0619 00:12:22.328992 27472 raft_consensus.cc:2487] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I0619 00:12:22.326606 27195 consensus_queue.cc:227] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.329815 26533 catalog_manager.cc:3786] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:22.341194 27199 ts_tablet_manager.cc:1403] T f4049071d2db4c629a9450fc85832e37 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
W0619 00:12:22.342543 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.345448 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.345777 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.328348 27513 raft_consensus.cc:479] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.346473 27513 leader_election.cc:243] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.347223 27197 tablet_bootstrap.cc:427] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:22.347470 27513 leader_election.cc:243] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.347605 27197 ts_tablet_manager.cc:1045] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.044s	user 0.004s	sys 0.001s
I0619 00:12:22.348093 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1e1883f069634e849dcba5e44201993e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:22.348510 27196 tablet_bootstrap.cc:427] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:22.348521 26949 raft_consensus.cc:2784] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.348788 26949 raft_consensus.cc:2732] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.815s (vote granted)
I0619 00:12:22.348919 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1e1883f069634e849dcba5e44201993e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:22.349339 27083 raft_consensus.cc:2784] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.349602 27083 raft_consensus.cc:2732] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.743s (vote granted)
I0619 00:12:22.350776 27196 tablet_bootstrap.cc:578] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:22.352576 27200 ts_tablet_manager.cc:1382] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.353480 26949 raft_consensus.cc:2245] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:22.354279 27083 raft_consensus.cc:2245] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:22.354502 26732 leader_election.cc:379] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.353637 27200 log.cc:986] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/f4049071d2db4c629a9450fc85832e37
I0619 00:12:22.354816 26732 leader_election.cc:270] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:22.355108 27200 ts_tablet_manager.cc:1403] T f4049071d2db4c629a9450fc85832e37 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:22.355082 27197 raft_consensus.cc:325] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.355523 26732 leader_election.cc:379] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.355733 27197 raft_consensus.cc:351] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.355911 27197 raft_consensus.cc:628] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:22.356025 27513 raft_consensus.cc:2732] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.785s (election complete)
I0619 00:12:22.342062 27195 ts_tablet_manager.cc:1073] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.001s
I0619 00:12:22.356302 27513 raft_consensus.cc:2539] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:22.356418 27197 consensus_queue.cc:227] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.357147 27195 ts_tablet_manager.cc:1028] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:22.357962 27197 ts_tablet_manager.cc:1073] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
W0619 00:12:22.358809 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.356616 27513 raft_consensus.cc:593] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:22.360406 27513 consensus_queue.cc:204] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.361433 27195 tablet_bootstrap.cc:427] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:22.361927 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.363252 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet f4049071d2db4c629a9450fc85832e37 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:22.367305 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet f4049071d2db4c629a9450fc85832e37 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:22.369601 27195 tablet_bootstrap.cc:578] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:22.379297 26533 catalog_manager.cc:3786] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:22.386888 27195 tablet_bootstrap.cc:427] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:22.387228 27195 ts_tablet_manager.cc:1045] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.026s	user 0.005s	sys 0.002s
I0619 00:12:22.388772 27195 raft_consensus.cc:325] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.393409 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet ca65280877d540f4aae9152095741d83 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.389430 27195 raft_consensus.cc:351] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.396461 27195 raft_consensus.cc:628] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:22.400266 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 1e1883f069634e849dcba5e44201993e from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.404848 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet ca65280877d540f4aae9152095741d83
I0619 00:12:22.405129 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ca65280877d540f4aae9152095741d83 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:22.405501 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ca65280877d540f4aae9152095741d83 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:22.405897 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ca65280877d540f4aae9152095741d83 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:22.407132 27062 tablet_service.cc:800] Processing DeleteTablet for tablet ca65280877d540f4aae9152095741d83 with delete_type TABLET_DATA_DELETED (replaced by tablet 917aeb5b70af43b5870c60c715e7cbb4) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:22.407580 26929 tablet_service.cc:800] Processing DeleteTablet for tablet ca65280877d540f4aae9152095741d83 with delete_type TABLET_DATA_DELETED (replaced by tablet 917aeb5b70af43b5870c60c715e7cbb4) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:22.408074 27210 tablet_replica.cc:261] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:22.408097 27199 tablet_replica.cc:261] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:22.408316 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.408316 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.408515 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.408515 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.408720 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.408720 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.408905 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.408910 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.409124 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.409124 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.409155 26532 catalog_manager.cc:4596] ReplaceTablet: tablet ca65280877d540f4aae9152095741d83 deleted and replaced by tablet 917aeb5b70af43b5870c60c715e7cbb4
I0619 00:12:22.409315 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.409312 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.409493 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.409507 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.409673 27199 raft_consensus.cc:2042] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.409687 27210 raft_consensus.cc:2042] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.409955 27210 raft_consensus.cc:2069] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.409955 27199 raft_consensus.cc:2069] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.410966 26792 tablet_service.cc:800] Processing DeleteTablet for tablet ca65280877d540f4aae9152095741d83 with delete_type TABLET_DATA_DELETED (replaced by tablet 917aeb5b70af43b5870c60c715e7cbb4) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:22.411432 27200 tablet_replica.cc:261] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:22.411664 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.411869 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.411906 27199 ts_tablet_manager.cc:1369] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.412052 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.412235 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.412446 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.412626 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.412830 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(ca65280877d540f4aae9152095741d83)
I0619 00:12:22.412995 27200 raft_consensus.cc:2042] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:22.413192 27200 consensus_peers.cc:456] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:22.413389 27200 consensus_peers.cc:456] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:22.413378 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 1e1883f069634e849dcba5e44201993e
I0619 00:12:22.413678 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1e1883f069634e849dcba5e44201993e on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.413856 27196 tablet_bootstrap.cc:427] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:22.414052 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1e1883f069634e849dcba5e44201993e on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.414259 27196 ts_tablet_manager.cc:1045] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.096s	user 0.007s	sys 0.003s
I0619 00:12:22.414989 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1e1883f069634e849dcba5e44201993e on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.415088 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 1e1883f069634e849dcba5e44201993e with delete_type TABLET_DATA_DELETED (replaced by tablet 7fe8dd51fc2c448c9e1e357c24d67104) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:22.415923 27210 ts_tablet_manager.cc:1369] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.415917 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 1e1883f069634e849dcba5e44201993e with delete_type TABLET_DATA_DELETED (replaced by tablet 7fe8dd51fc2c448c9e1e357c24d67104) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:22.416131 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 1e1883f069634e849dcba5e44201993e deleted and replaced by tablet 7fe8dd51fc2c448c9e1e357c24d67104
I0619 00:12:22.413871 27200 raft_consensus.cc:2069] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.416519 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 1e1883f069634e849dcba5e44201993e with delete_type TABLET_DATA_DELETED (replaced by tablet 7fe8dd51fc2c448c9e1e357c24d67104) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:22.418874 27200 ts_tablet_manager.cc:1369] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.397608 27195 consensus_queue.cc:227] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.424237 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.427181 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.432837 26929 tablet_service.cc:759] Processing CreateTablet for tablet 917aeb5b70af43b5870c60c715e7cbb4 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.433439 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 917aeb5b70af43b5870c60c715e7cbb4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.433339 26792 tablet_service.cc:759] Processing CreateTablet for tablet 917aeb5b70af43b5870c60c715e7cbb4 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.433948 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 917aeb5b70af43b5870c60c715e7cbb4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.434844 27062 tablet_service.cc:759] Processing CreateTablet for tablet 917aeb5b70af43b5870c60c715e7cbb4 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.435425 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 917aeb5b70af43b5870c60c715e7cbb4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.436769 27195 ts_tablet_manager.cc:1073] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.049s	user 0.004s	sys 0.000s
I0619 00:12:22.437463 27210 ts_tablet_manager.cc:1382] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.437727 27210 log.cc:986] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/ca65280877d540f4aae9152095741d83
I0619 00:12:22.441000 27199 ts_tablet_manager.cc:1382] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.441257 27199 log.cc:986] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/ca65280877d540f4aae9152095741d83
I0619 00:12:22.441916 27199 ts_tablet_manager.cc:1403] T ca65280877d540f4aae9152095741d83 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:22.444875 26791 tablet_service.cc:759] Processing CreateTablet for tablet 7fe8dd51fc2c448c9e1e357c24d67104 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.445451 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7fe8dd51fc2c448c9e1e357c24d67104. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.445704 26928 tablet_service.cc:759] Processing CreateTablet for tablet 7fe8dd51fc2c448c9e1e357c24d67104 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.446287 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7fe8dd51fc2c448c9e1e357c24d67104. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.446732 26792 ts_tablet_manager.cc:1173] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.429260 27196 raft_consensus.cc:325] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.450103 26929 ts_tablet_manager.cc:1173] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.450372 27196 raft_consensus.cc:351] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.450495 27197 ts_tablet_manager.cc:1028] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:22.450856 27197 tablet_bootstrap.cc:427] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:22.455013 27197 tablet_bootstrap.cc:578] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:22.455220 27196 raft_consensus.cc:628] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:22.455788 27196 consensus_queue.cc:227] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.460510 26791 ts_tablet_manager.cc:1173] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.465337 26928 ts_tablet_manager.cc:1173] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:22.460934 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.470543 27210 ts_tablet_manager.cc:1403] T ca65280877d540f4aae9152095741d83 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
W0619 00:12:22.472311 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.461975 27196 ts_tablet_manager.cc:1073] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.036s	user 0.003s	sys 0.001s
I0619 00:12:22.472441 27210 tablet_replica.cc:261] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:22.472693 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.472604 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet ca65280877d540f4aae9152095741d83 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:22.472729 27196 ts_tablet_manager.cc:1028] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:22.472882 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.473057 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.473069 27196 tablet_bootstrap.cc:427] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:22.473246 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.473457 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.473656 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.473829 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.473991 27210 raft_consensus.cc:2042] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.474236 27210 raft_consensus.cc:2069] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.448963 27063 tablet_service.cc:759] Processing CreateTablet for tablet 7fe8dd51fc2c448c9e1e357c24d67104 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.452641 27062 ts_tablet_manager.cc:1173] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.475406 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7fe8dd51fc2c448c9e1e357c24d67104. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.475425 27196 tablet_bootstrap.cc:578] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:22.469168 27200 ts_tablet_manager.cc:1382] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.477154 27200 log.cc:986] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/ca65280877d540f4aae9152095741d83
I0619 00:12:22.477741 27200 ts_tablet_manager.cc:1403] T ca65280877d540f4aae9152095741d83 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:22.478713 27195 ts_tablet_manager.cc:1028] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:22.479092 27195 tablet_bootstrap.cc:427] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:22.479471 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet ca65280877d540f4aae9152095741d83 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:22.479987 27199 tablet_replica.cc:261] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:22.480090 27200 tablet_replica.cc:261] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:22.480201 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480310 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480381 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480495 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480545 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480692 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480738 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480870 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.480921 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.481078 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.481081 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.481241 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.481277 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.481411 27199 raft_consensus.cc:2042] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.481451 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(1e1883f069634e849dcba5e44201993e)
I0619 00:12:22.481619 27200 raft_consensus.cc:2042] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:22.481679 27199 raft_consensus.cc:2069] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.481838 27200 consensus_peers.cc:456] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:22.482038 27200 consensus_peers.cc:456] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:22.482076 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet ca65280877d540f4aae9152095741d83 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:22.482372 27200 pending_rounds.cc:63] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:22.482532 27200 pending_rounds.cc:68] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391034319134720 op_type: NO_OP noop_request { }
I0619 00:12:22.483264 27200 raft_consensus.cc:2622] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:22.483714 27197 tablet_bootstrap.cc:427] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:22.484055 27197 ts_tablet_manager.cc:1045] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.033s	user 0.003s	sys 0.003s
I0619 00:12:22.484272 27200 raft_consensus.cc:2069] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.485400 27197 raft_consensus.cc:325] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.481313 27195 tablet_bootstrap.cc:578] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:22.486023 27197 raft_consensus.cc:351] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.486196 27197 raft_consensus.cc:628] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:22.487017 27197 consensus_queue.cc:227] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.488416 27197 ts_tablet_manager.cc:1073] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:12:22.488768 27197 ts_tablet_manager.cc:1028] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:22.489100 27197 tablet_bootstrap.cc:427] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
W0619 00:12:22.489382 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.490067 27200 ts_tablet_manager.cc:1369] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.494333 27196 tablet_bootstrap.cc:427] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:22.494717 27196 ts_tablet_manager.cc:1045] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.000s
I0619 00:12:22.495164 27210 ts_tablet_manager.cc:1369] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.496191 27196 raft_consensus.cc:325] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.496814 27063 ts_tablet_manager.cc:1173] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.496856 27196 raft_consensus.cc:351] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.497038 27196 raft_consensus.cc:628] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:22.497454 27196 consensus_queue.cc:227] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.499413 27197 tablet_bootstrap.cc:578] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:22.501135 27199 ts_tablet_manager.cc:1369] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:22.501158 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.502800 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.503587 27196 ts_tablet_manager.cc:1073] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I0619 00:12:22.503983 27196 ts_tablet_manager.cc:1028] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:22.504319 27196 tablet_bootstrap.cc:427] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:22.506460 27196 tablet_bootstrap.cc:578] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:22.511378 27195 tablet_bootstrap.cc:427] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:22.511721 27195 ts_tablet_manager.cc:1045] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.033s	user 0.004s	sys 0.002s
I0619 00:12:22.512953 27195 raft_consensus.cc:325] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.513550 27195 raft_consensus.cc:351] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.513763 27195 raft_consensus.cc:628] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:22.514344 27195 consensus_queue.cc:227] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.517455 27197 tablet_bootstrap.cc:427] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:22.517817 27197 ts_tablet_manager.cc:1045] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.029s	user 0.005s	sys 0.000s
I0619 00:12:22.519174 27197 raft_consensus.cc:325] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.519824 27197 raft_consensus.cc:351] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.520020 27197 raft_consensus.cc:628] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
W0619 00:12:22.520793 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.520457 27197 consensus_queue.cc:227] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.522434 27197 ts_tablet_manager.cc:1073] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
W0619 00:12:22.522955 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.524698 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.526998 27200 ts_tablet_manager.cc:1382] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.527693 27210 ts_tablet_manager.cc:1382] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.529994 27199 ts_tablet_manager.cc:1382] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.531225 27200 log.cc:986] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/1e1883f069634e849dcba5e44201993e
I0619 00:12:22.531488 27199 log.cc:986] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/1e1883f069634e849dcba5e44201993e
I0619 00:12:22.531746 27210 log.cc:986] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/1e1883f069634e849dcba5e44201993e
I0619 00:12:22.532754 27200 ts_tablet_manager.cc:1403] T 1e1883f069634e849dcba5e44201993e P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:22.533427 27199 ts_tablet_manager.cc:1403] T 1e1883f069634e849dcba5e44201993e P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:22.533685 27210 ts_tablet_manager.cc:1403] T 1e1883f069634e849dcba5e44201993e P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
W0619 00:12:22.533476 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.534384 27195 ts_tablet_manager.cc:1073] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.000s
I0619 00:12:22.534967 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 1e1883f069634e849dcba5e44201993e (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.535153 27196 tablet_bootstrap.cc:427] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:22.535471 27196 ts_tablet_manager.cc:1045] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.031s	user 0.005s	sys 0.000s
I0619 00:12:22.535636 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 1e1883f069634e849dcba5e44201993e (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.536273 27195 ts_tablet_manager.cc:1028] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:22.536665 27195 tablet_bootstrap.cc:427] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:22.536937 27196 raft_consensus.cc:325] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.537835 27196 raft_consensus.cc:351] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.538094 27196 raft_consensus.cc:628] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:22.539100 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 1e1883f069634e849dcba5e44201993e (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.538630 27196 consensus_queue.cc:227] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
W0619 00:12:22.541823 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.542423 27513 raft_consensus.cc:457] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:22.543094 27513 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.816s (starting election)
I0619 00:12:22.543328 27513 raft_consensus.cc:479] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.543916 27195 tablet_bootstrap.cc:578] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:22.544243 27513 leader_election.cc:243] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.544621 27513 leader_election.cc:243] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.546267 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "917aeb5b70af43b5870c60c715e7cbb4" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:22.546365 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "917aeb5b70af43b5870c60c715e7cbb4" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:22.546780 26949 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.242s (vote granted)
I0619 00:12:22.546857 27083 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.612s (vote granted)
I0619 00:12:22.547003 26949 raft_consensus.cc:2245] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:22.547080 27083 raft_consensus.cc:2245] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
W0619 00:12:22.547219 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.548069 27196 ts_tablet_manager.cc:1073] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I0619 00:12:22.548243 26732 leader_election.cc:379] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.548503 26732 leader_election.cc:270] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:22.548948 27513 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.517s (election complete)
I0619 00:12:22.549175 27513 raft_consensus.cc:2539] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:22.549393 27513 raft_consensus.cc:457] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:22.549597 27513 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.128s (starting election)
I0619 00:12:22.549245 26732 leader_election.cc:379] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.549826 27513 raft_consensus.cc:2784] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.554328 27513 raft_consensus.cc:479] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.555286 27513 leader_election.cc:243] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.556735 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "917aeb5b70af43b5870c60c715e7cbb4" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:22.557101 27513 leader_election.cc:243] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.557137 27083 raft_consensus.cc:2784] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.557360 27083 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.589s (vote granted)
I0619 00:12:22.558511 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "917aeb5b70af43b5870c60c715e7cbb4" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:22.558928 26949 raft_consensus.cc:2784] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.559165 26949 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.155s (vote granted)
I0619 00:12:22.561671 27083 raft_consensus.cc:2245] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:22.562625 26732 leader_election.cc:379] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:22.562875 26732 leader_election.cc:270] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:22.563285 27195 tablet_bootstrap.cc:427] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:22.563361 27513 raft_consensus.cc:2732] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.650s (election complete)
I0619 00:12:22.563382 26949 raft_consensus.cc:2245] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:22.563659 27195 ts_tablet_manager.cc:1045] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.027s	user 0.005s	sys 0.000s
I0619 00:12:22.563625 27513 raft_consensus.cc:2539] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
W0619 00:12:22.565636 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.566208 26732 leader_election.cc:379] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.565955 27195 raft_consensus.cc:325] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.563969 27513 raft_consensus.cc:593] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:22.566618 27195 raft_consensus.cc:351] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.566797 27195 raft_consensus.cc:628] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:22.567032 27513 consensus_queue.cc:204] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
I0619 00:12:22.567327 27195 consensus_queue.cc:227] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } }
W0619 00:12:22.571061 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.579901 27195 ts_tablet_manager.cc:1073] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.000s
I0619 00:12:22.579247 26533 catalog_manager.cc:3786] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } }
W0619 00:12:22.580852 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.653045 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet fac1a493e4a245b2b5a017e61f877e4f from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.654875 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet fac1a493e4a245b2b5a017e61f877e4f from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.656697 27472 raft_consensus.cc:457] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:22.656972 27472 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.602s (starting election)
I0619 00:12:22.657223 27472 raft_consensus.cc:479] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.658211 27472 leader_election.cc:243] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.658635 27472 leader_election.cc:243] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.660241 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "272d0c23372d485595f52ce73d28f56c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:22.660758 26949 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.781s (vote granted)
I0619 00:12:22.660989 26949 raft_consensus.cc:2245] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:22.663336 27013 leader_election.cc:379] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.663566 27013 leader_election.cc:270] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:22.663134 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "272d0c23372d485595f52ce73d28f56c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:22.667289 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet fac1a493e4a245b2b5a017e61f877e4f
I0619 00:12:22.667590 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fac1a493e4a245b2b5a017e61f877e4f on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:22.667929 27472 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.555s (election complete)
I0619 00:12:22.668011 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fac1a493e4a245b2b5a017e61f877e4f on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:22.668174 27472 raft_consensus.cc:2539] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:22.668390 27472 raft_consensus.cc:457] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:22.668397 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fac1a493e4a245b2b5a017e61f877e4f on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:22.668622 27472 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.246s (starting election)
I0619 00:12:22.668730 26533 catalog_manager.cc:4596] ReplaceTablet: tablet fac1a493e4a245b2b5a017e61f877e4f deleted and replaced by tablet a39cff5a026248e2b668dbdde15b1d44
I0619 00:12:22.668831 27472 raft_consensus.cc:2784] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.671826 27063 tablet_service.cc:800] Processing DeleteTablet for tablet fac1a493e4a245b2b5a017e61f877e4f with delete_type TABLET_DATA_DELETED (replaced by tablet a39cff5a026248e2b668dbdde15b1d44) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:22.672279 26928 tablet_service.cc:800] Processing DeleteTablet for tablet fac1a493e4a245b2b5a017e61f877e4f with delete_type TABLET_DATA_DELETED (replaced by tablet a39cff5a026248e2b668dbdde15b1d44) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:22.672786 27199 tablet_replica.cc:261] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:22.673010 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.673188 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.673355 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.673526 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.673759 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.673946 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.674129 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.674309 27199 raft_consensus.cc:2042] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.674574 27199 raft_consensus.cc:2069] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.675741 27199 ts_tablet_manager.cc:1369] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.676828 27210 tablet_replica.cc:261] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:22.677070 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.677258 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.677444 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.677640 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.677845 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.678026 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.678208 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.678375 27210 raft_consensus.cc:2042] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:22.678577 27210 consensus_peers.cc:456] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:22.678808 27210 consensus_peers.cc:456] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:22.679155 27210 pending_rounds.cc:63] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:22.679291 27210 pending_rounds.cc:68] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391034161188864 op_type: NO_OP noop_request { }
I0619 00:12:22.679589 27210 raft_consensus.cc:2622] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:22.679352 27472 raft_consensus.cc:479] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.679816 27210 raft_consensus.cc:2069] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.680721 26791 tablet_service.cc:800] Processing DeleteTablet for tablet fac1a493e4a245b2b5a017e61f877e4f with delete_type TABLET_DATA_DELETED (replaced by tablet a39cff5a026248e2b668dbdde15b1d44) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:22.681921 27210 ts_tablet_manager.cc:1369] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.682076 27200 tablet_replica.cc:261] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:22.682303 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.682497 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.682705 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.682886 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.683094 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.683279 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.683454 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(fac1a493e4a245b2b5a017e61f877e4f)
I0619 00:12:22.683650 27200 raft_consensus.cc:2042] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.683928 27200 raft_consensus.cc:2069] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.685135 26791 tablet_service.cc:759] Processing CreateTablet for tablet a39cff5a026248e2b668dbdde15b1d44 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.686301 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a39cff5a026248e2b668dbdde15b1d44. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.686903 26928 tablet_service.cc:759] Processing CreateTablet for tablet a39cff5a026248e2b668dbdde15b1d44 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.687647 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a39cff5a026248e2b668dbdde15b1d44. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.690029 27063 tablet_service.cc:759] Processing CreateTablet for tablet a39cff5a026248e2b668dbdde15b1d44 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.690675 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a39cff5a026248e2b668dbdde15b1d44. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.664911 26811 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.543s (vote granted)
I0619 00:12:22.691767 26811 raft_consensus.cc:2245] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:22.693146 27015 leader_election.cc:379] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.698051 27472 leader_election.cc:243] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.699681 27200 ts_tablet_manager.cc:1369] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.700559 27472 leader_election.cc:243] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.701563 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "272d0c23372d485595f52ce73d28f56c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:22.702003 26811 raft_consensus.cc:2784] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.702055 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "272d0c23372d485595f52ce73d28f56c" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:22.702394 26811 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.746s (vote granted)
I0619 00:12:22.702519 26949 raft_consensus.cc:2784] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.702790 26949 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.976s (vote granted)
I0619 00:12:22.706908 26949 raft_consensus.cc:2245] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:22.707897 27013 leader_election.cc:379] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.708106 27013 leader_election.cc:270] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:22.708683 27472 raft_consensus.cc:2732] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.127s (election complete)
I0619 00:12:22.708920 27472 raft_consensus.cc:2539] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:22.713544 27472 raft_consensus.cc:593] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:22.714046 27472 consensus_queue.cc:204] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.716061 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.716791 27210 ts_tablet_manager.cc:1382] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:22.717044 27210 log.cc:986] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/fac1a493e4a245b2b5a017e61f877e4f
I0619 00:12:22.717638 27210 ts_tablet_manager.cc:1403] T fac1a493e4a245b2b5a017e61f877e4f P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:22.719461 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet fac1a493e4a245b2b5a017e61f877e4f (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:22.721645 26811 raft_consensus.cc:2245] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:22.722869 27015 leader_election.cc:379] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.726876 26791 ts_tablet_manager.cc:1173] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.727449 27063 ts_tablet_manager.cc:1173] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.727866 27195 ts_tablet_manager.cc:1028] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:22.728209 27195 tablet_bootstrap.cc:427] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:22.730829 27195 tablet_bootstrap.cc:578] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:22.730810 26928 ts_tablet_manager.cc:1173] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.731173 27197 ts_tablet_manager.cc:1028] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:22.731526 27197 tablet_bootstrap.cc:427] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:22.734248 27197 tablet_bootstrap.cc:578] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:22.739703 26532 catalog_manager.cc:3786] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:22.744094 27196 ts_tablet_manager.cc:1028] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:22.744485 27196 tablet_bootstrap.cc:427] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:22.748270 27199 ts_tablet_manager.cc:1382] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.748574 27199 log.cc:986] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/fac1a493e4a245b2b5a017e61f877e4f
I0619 00:12:22.749454 27197 tablet_bootstrap.cc:427] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:22.749896 27197 ts_tablet_manager.cc:1045] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.002s
I0619 00:12:22.750133 27196 tablet_bootstrap.cc:578] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:22.752082 27197 raft_consensus.cc:325] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.752768 27197 raft_consensus.cc:351] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.752974 27197 raft_consensus.cc:628] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:22.753458 27197 consensus_queue.cc:227] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.755928 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.756593 27197 ts_tablet_manager.cc:1073] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.002s
I0619 00:12:22.757299 27199 ts_tablet_manager.cc:1403] T fac1a493e4a245b2b5a017e61f877e4f P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
W0619 00:12:22.757458 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.757767 26523 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918): Couldn't send request to peer 555bff44d4f942eaad13f790cca0a696 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times.
I0619 00:12:22.760787 27200 ts_tablet_manager.cc:1382] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.761728 27200 log.cc:986] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/fac1a493e4a245b2b5a017e61f877e4f
I0619 00:12:22.761937 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet fac1a493e4a245b2b5a017e61f877e4f (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:22.762295 27200 ts_tablet_manager.cc:1403] T fac1a493e4a245b2b5a017e61f877e4f P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:22.771790 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet fac1a493e4a245b2b5a017e61f877e4f (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:22.773150 27196 tablet_bootstrap.cc:427] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:22.773465 27196 ts_tablet_manager.cc:1045] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.029s	user 0.005s	sys 0.008s
I0619 00:12:22.775738 27196 raft_consensus.cc:325] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.776341 27196 raft_consensus.cc:351] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.776526 27196 raft_consensus.cc:628] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:22.776991 27196 consensus_queue.cc:227] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.778555 27195 tablet_bootstrap.cc:427] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:22.778944 27195 ts_tablet_manager.cc:1045] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.051s	user 0.006s	sys 0.008s
W0619 00:12:22.782127 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.783289 27195 raft_consensus.cc:325] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.783952 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.785001 27196 ts_tablet_manager.cc:1073] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.001s
I0619 00:12:22.784046 27195 raft_consensus.cc:351] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.786373 27195 raft_consensus.cc:628] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:22.786893 27195 consensus_queue.cc:227] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.795776 27472 raft_consensus.cc:457] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:22.795991 27472 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.962s (starting election)
I0619 00:12:22.796228 27472 raft_consensus.cc:479] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.797472 27472 leader_election.cc:243] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.797889 27472 leader_election.cc:243] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
W0619 00:12:22.799124 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.800076 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a39cff5a026248e2b668dbdde15b1d44" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:22.800101 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a39cff5a026248e2b668dbdde15b1d44" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:22.800524 26811 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.816s (vote granted)
I0619 00:12:22.800601 26949 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.711s (vote granted)
I0619 00:12:22.800770 26811 raft_consensus.cc:2245] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:22.800849 26949 raft_consensus.cc:2245] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:22.801939 27013 leader_election.cc:379] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.802199 27013 leader_election.cc:270] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:22.803205 27472 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.944s (election complete)
I0619 00:12:22.803303 27015 leader_election.cc:379] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.803465 27472 raft_consensus.cc:2539] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:22.803704 27472 raft_consensus.cc:457] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:22.803899 27472 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.136s (starting election)
I0619 00:12:22.804092 27472 raft_consensus.cc:2784] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
W0619 00:12:22.805833 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.806735 27195 ts_tablet_manager.cc:1073] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.001s
I0619 00:12:22.808560 27472 raft_consensus.cc:479] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.809525 27472 leader_election.cc:243] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.809574 27474 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536
I0619 00:12:22.814129 27472 leader_election.cc:243] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.814705 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a39cff5a026248e2b668dbdde15b1d44" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:22.815093 26949 raft_consensus.cc:2784] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.815325 26949 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.638s (vote granted)
I0619 00:12:22.815663 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a39cff5a026248e2b668dbdde15b1d44" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:22.816053 26811 raft_consensus.cc:2784] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:22.816279 26811 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.919s (vote granted)
I0619 00:12:22.819977 26949 raft_consensus.cc:2245] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:22.820951 27013 leader_election.cc:379] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:22.821177 27013 leader_election.cc:270] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:22.820459 26811 raft_consensus.cc:2245] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:22.821949 27472 raft_consensus.cc:2732] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.795s (election complete)
I0619 00:12:22.822178 27472 raft_consensus.cc:2539] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:22.822469 27472 raft_consensus.cc:593] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:22.822918 27472 consensus_queue.cc:204] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.822559 27015 leader_election.cc:379] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:22.825175 27474 master_main.cc:83] Starting Master server...
W0619 00:12:22.825259 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.840720 26532 catalog_manager.cc:3786] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:22.880249 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 7fe8dd51fc2c448c9e1e357c24d67104 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.881384 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 7fe8dd51fc2c448c9e1e357c24d67104 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:22.895789 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 7fe8dd51fc2c448c9e1e357c24d67104
I0619 00:12:22.896091 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7fe8dd51fc2c448c9e1e357c24d67104 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:22.896464 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7fe8dd51fc2c448c9e1e357c24d67104 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:22.896842 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7fe8dd51fc2c448c9e1e357c24d67104 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:22.897136 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 7fe8dd51fc2c448c9e1e357c24d67104 deleted and replaced by tablet a4028407d91d4bc09b5d967dee87110b
I0619 00:12:22.898468 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 7fe8dd51fc2c448c9e1e357c24d67104 with delete_type TABLET_DATA_DELETED (replaced by tablet a4028407d91d4bc09b5d967dee87110b) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:22.898963 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 7fe8dd51fc2c448c9e1e357c24d67104 with delete_type TABLET_DATA_DELETED (replaced by tablet a4028407d91d4bc09b5d967dee87110b) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:22.899264 27200 tablet_replica.cc:261] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:22.899487 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.899485 27199 tablet_replica.cc:261] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:22.899689 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.899729 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.899869 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.899912 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900020 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900082 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900234 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900244 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900418 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900439 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900568 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900610 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900738 27200 raft_consensus.cc:2042] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.900800 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.900956 27199 raft_consensus.cc:2042] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.900984 27200 raft_consensus.cc:2069] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.901199 27199 raft_consensus.cc:2069] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.901510 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 7fe8dd51fc2c448c9e1e357c24d67104 with delete_type TABLET_DATA_DELETED (replaced by tablet a4028407d91d4bc09b5d967dee87110b) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:22.902040 27210 tablet_replica.cc:261] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:22.902264 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.902448 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.902487 27199 ts_tablet_manager.cc:1369] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.902628 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.902834 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.903039 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.903228 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.903391 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(7fe8dd51fc2c448c9e1e357c24d67104)
I0619 00:12:22.903554 27210 raft_consensus.cc:2042] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:22.903776 27200 ts_tablet_manager.cc:1369] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.903833 27210 raft_consensus.cc:2069] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:22.905719 27210 ts_tablet_manager.cc:1369] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:22.916441 26791 tablet_service.cc:759] Processing CreateTablet for tablet a4028407d91d4bc09b5d967dee87110b (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.917081 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4028407d91d4bc09b5d967dee87110b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.917922 27063 tablet_service.cc:759] Processing CreateTablet for tablet a4028407d91d4bc09b5d967dee87110b (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.918545 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4028407d91d4bc09b5d967dee87110b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.922050 26928 tablet_service.cc:759] Processing CreateTablet for tablet a4028407d91d4bc09b5d967dee87110b (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:22.922688 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a4028407d91d4bc09b5d967dee87110b. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:22.929389 26791 ts_tablet_manager.cc:1173] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:22.930258 27196 ts_tablet_manager.cc:1028] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:22.930639 27196 tablet_bootstrap.cc:427] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:22.932970 27196 tablet_bootstrap.cc:578] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:22.934672 27210 ts_tablet_manager.cc:1382] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.938971 27196 tablet_bootstrap.cc:427] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:22.939410 27196 ts_tablet_manager.cc:1045] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.001s
I0619 00:12:22.939985 27200 ts_tablet_manager.cc:1382] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.940256 27210 log.cc:986] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/7fe8dd51fc2c448c9e1e357c24d67104
I0619 00:12:22.940814 27210 ts_tablet_manager.cc:1403] T 7fe8dd51fc2c448c9e1e357c24d67104 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:22.943145 27200 log.cc:986] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/7fe8dd51fc2c448c9e1e357c24d67104
I0619 00:12:22.944638 27199 ts_tablet_manager.cc:1382] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:22.944769 27200 ts_tablet_manager.cc:1403] T 7fe8dd51fc2c448c9e1e357c24d67104 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:22.944887 27199 log.cc:986] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/7fe8dd51fc2c448c9e1e357c24d67104
I0619 00:12:22.940655 27196 raft_consensus.cc:325] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.945744 27196 raft_consensus.cc:351] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.945974 27196 raft_consensus.cc:628] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:22.946099 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 7fe8dd51fc2c448c9e1e357c24d67104 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.946483 27196 consensus_queue.cc:227] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.949101 26928 ts_tablet_manager.cc:1173] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:22.949138 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.949335 27063 ts_tablet_manager.cc:1173] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:22.951006 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.951716 27197 ts_tablet_manager.cc:1028] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:22.951932 27196 ts_tablet_manager.cc:1073] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.000s
I0619 00:12:22.952067 27197 tablet_bootstrap.cc:427] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:22.952102 27195 ts_tablet_manager.cc:1028] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:22.952443 27195 tablet_bootstrap.cc:427] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:22.954921 27197 tablet_bootstrap.cc:578] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:22.957371 27195 tablet_bootstrap.cc:578] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:22.958370 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 7fe8dd51fc2c448c9e1e357c24d67104 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.959034 27199 ts_tablet_manager.cc:1403] T 7fe8dd51fc2c448c9e1e357c24d67104 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:22.960811 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 7fe8dd51fc2c448c9e1e357c24d67104 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:22.961346 27197 tablet_bootstrap.cc:427] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:22.961652 27197 ts_tablet_manager.cc:1045] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.001s
I0619 00:12:22.962879 27197 raft_consensus.cc:325] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.963459 27197 raft_consensus.cc:351] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.963660 27197 raft_consensus.cc:628] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:22.964066 27197 consensus_queue.cc:227] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.966675 27197 ts_tablet_manager.cc:1073] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:12:22.967896 26949 raft_consensus.cc:1081] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:22.968411 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.970839 27195 tablet_bootstrap.cc:427] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:22.971609 27195 ts_tablet_manager.cc:1045] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.005s
W0619 00:12:22.972193 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.972357 27513 consensus_queue.cc:963] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I0619 00:12:22.972993 27195 raft_consensus.cc:325] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:22.973640 27195 raft_consensus.cc:351] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:22.973868 27195 raft_consensus.cc:628] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:22.974303 27195 consensus_queue.cc:227] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:22.977329 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.979003 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.981118 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.984057 27195 ts_tablet_manager.cc:1073] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I0619 00:12:22.993552 27083 raft_consensus.cc:1081] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:22.995352 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:22.996464 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:22.997340 27424 consensus_queue.cc:963] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:23.002110 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:23.013741 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.078307 27513 raft_consensus.cc:457] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:23.078572 27513 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.119s (starting election)
I0619 00:12:23.078856 27513 raft_consensus.cc:479] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.079898 27513 leader_election.cc:243] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.080302 27513 leader_election.cc:243] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.082007 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4028407d91d4bc09b5d967dee87110b" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:23.082499 27083 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.540s (vote granted)
I0619 00:12:23.082752 27083 raft_consensus.cc:2245] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:23.082679 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4028407d91d4bc09b5d967dee87110b" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:23.083166 26949 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.968s (vote granted)
I0619 00:12:23.083389 26949 raft_consensus.cc:2245] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:23.083915 26732 leader_election.cc:379] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.084193 26732 leader_election.cc:270] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:23.084743 27513 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.204s (election complete)
I0619 00:12:23.084956 27513 raft_consensus.cc:2539] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:23.085136 27513 raft_consensus.cc:457] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:23.085309 27513 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.221s (starting election)
I0619 00:12:23.085475 26732 leader_election.cc:379] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.085485 27513 raft_consensus.cc:2784] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.090991 27513 raft_consensus.cc:479] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.094048 27513 leader_election.cc:243] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.094419 27513 leader_election.cc:243] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.095942 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4028407d91d4bc09b5d967dee87110b" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:23.096015 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a4028407d91d4bc09b5d967dee87110b" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:23.096354 27083 raft_consensus.cc:2784] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.096432 26949 raft_consensus.cc:2784] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.096621 27083 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.117s (vote granted)
I0619 00:12:23.096693 26949 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.893s (vote granted)
I0619 00:12:23.100837 27083 raft_consensus.cc:2245] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:23.101034 26949 raft_consensus.cc:2245] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:23.102311 26732 leader_election.cc:379] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.102572 26732 leader_election.cc:270] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:23.103210 26732 leader_election.cc:379] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.104574 27513 raft_consensus.cc:2732] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.965s (election complete)
I0619 00:12:23.105314 26949 raft_consensus.cc:1081] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:23.107256 27472 consensus_queue.cc:963] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:23.107617 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 917aeb5b70af43b5870c60c715e7cbb4 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:23.105338 27513 raft_consensus.cc:2539] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:23.111038 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 917aeb5b70af43b5870c60c715e7cbb4 from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:23.112243 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:23.112282 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.112509 27513 raft_consensus.cc:593] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:23.113023 27513 consensus_queue.cc:204] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.120436 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 917aeb5b70af43b5870c60c715e7cbb4
I0619 00:12:23.120744 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 917aeb5b70af43b5870c60c715e7cbb4 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.121134 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 917aeb5b70af43b5870c60c715e7cbb4 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.121510 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 917aeb5b70af43b5870c60c715e7cbb4 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.121842 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 917aeb5b70af43b5870c60c715e7cbb4 deleted and replaced by tablet 40f4b63fe7a046ea95df101ba2aed54c
I0619 00:12:23.122746 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 917aeb5b70af43b5870c60c715e7cbb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 40f4b63fe7a046ea95df101ba2aed54c) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:23.123225 27200 tablet_replica.cc:261] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:23.123453 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.123673 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.123857 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.124038 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.123994 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 917aeb5b70af43b5870c60c715e7cbb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 40f4b63fe7a046ea95df101ba2aed54c) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:23.124230 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.124423 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.124596 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.124794 27200 raft_consensus.cc:2042] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:23.124982 27200 consensus_peers.cc:456] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:23.125195 27200 consensus_peers.cc:456] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:23.125727 27200 raft_consensus.cc:2069] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.125732 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 917aeb5b70af43b5870c60c715e7cbb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 40f4b63fe7a046ea95df101ba2aed54c) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.126256 27199 tablet_replica.cc:261] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:23.126498 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.126709 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.126878 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.127048 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.127257 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.127436 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.127605 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.127792 27199 raft_consensus.cc:2042] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.127972 27200 ts_tablet_manager.cc:1369] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.128049 27199 raft_consensus.cc:2069] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.128546 27210 tablet_replica.cc:261] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:23.128801 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.128989 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.129161 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.129333 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.129537 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.129747 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.129933 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(917aeb5b70af43b5870c60c715e7cbb4)
I0619 00:12:23.130116 27210 raft_consensus.cc:2042] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.130434 27210 raft_consensus.cc:2069] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.133569 27199 ts_tablet_manager.cc:1369] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.134037 27210 ts_tablet_manager.cc:1369] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.140795 26791 tablet_service.cc:759] Processing CreateTablet for tablet 40f4b63fe7a046ea95df101ba2aed54c (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.141414 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 40f4b63fe7a046ea95df101ba2aed54c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.141773 27063 tablet_service.cc:759] Processing CreateTablet for tablet 40f4b63fe7a046ea95df101ba2aed54c (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.142379 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 40f4b63fe7a046ea95df101ba2aed54c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.145180 27200 ts_tablet_manager.cc:1382] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.145314 26928 tablet_service.cc:759] Processing CreateTablet for tablet 40f4b63fe7a046ea95df101ba2aed54c (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.145922 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 40f4b63fe7a046ea95df101ba2aed54c. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.148681 27200 log.cc:986] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/917aeb5b70af43b5870c60c715e7cbb4
I0619 00:12:23.150853 27200 ts_tablet_manager.cc:1403] T 917aeb5b70af43b5870c60c715e7cbb4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:23.151197 27199 ts_tablet_manager.cc:1382] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.151427 27199 log.cc:986] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/917aeb5b70af43b5870c60c715e7cbb4
I0619 00:12:23.156349 26791 ts_tablet_manager.cc:1173] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.156498 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 917aeb5b70af43b5870c60c715e7cbb4 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:23.156960 27196 ts_tablet_manager.cc:1028] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:23.157310 27196 tablet_bootstrap.cc:427] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
W0619 00:12:23.159925 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.160073 26811 raft_consensus.cc:1081] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:23.163156 27546 consensus_queue.cc:963] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:23.164134 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.166642 26532 catalog_manager.cc:3786] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:23.168170 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
W0619 00:12:23.168856 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.172224 27199 ts_tablet_manager.cc:1403] T 917aeb5b70af43b5870c60c715e7cbb4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:23.174306 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 917aeb5b70af43b5870c60c715e7cbb4 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:23.174502 27063 ts_tablet_manager.cc:1173] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:23.174453 26523 consensus_peers.cc:434] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 -> Peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918): Couldn't send request to peer 555bff44d4f942eaad13f790cca0a696 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times.
I0619 00:12:23.174887 27195 ts_tablet_manager.cc:1028] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:23.175235 27195 tablet_bootstrap.cc:427] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:23.176820 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.178721 27195 tablet_bootstrap.cc:578] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:23.185624 26928 ts_tablet_manager.cc:1173] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.186056 27195 tablet_bootstrap.cc:427] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:23.186353 27195 ts_tablet_manager.cc:1045] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.002s
I0619 00:12:23.187865 27196 tablet_bootstrap.cc:578] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:23.187669 27195 raft_consensus.cc:325] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.188282 27195 raft_consensus.cc:351] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.188473 27195 raft_consensus.cc:628] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:23.188971 27210 ts_tablet_manager.cc:1382] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.189025 27197 ts_tablet_manager.cc:1028] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:23.188920 27195 consensus_queue.cc:227] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.189477 27197 tablet_bootstrap.cc:427] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:23.190352 27195 ts_tablet_manager.cc:1073] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:12:23.191699 27197 tablet_bootstrap.cc:578] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
W0619 00:12:23.191890 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:23.193805 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.197986 27210 log.cc:986] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/917aeb5b70af43b5870c60c715e7cbb4
I0619 00:12:23.198598 27210 ts_tablet_manager.cc:1403] T 917aeb5b70af43b5870c60c715e7cbb4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:23.202517 27196 tablet_bootstrap.cc:427] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:23.202884 27196 ts_tablet_manager.cc:1045] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.046s	user 0.004s	sys 0.007s
I0619 00:12:23.204434 27196 raft_consensus.cc:325] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.205070 27196 raft_consensus.cc:351] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.205271 27196 raft_consensus.cc:628] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:23.206020 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 917aeb5b70af43b5870c60c715e7cbb4 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:23.205827 27196 consensus_queue.cc:227] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.208842 27197 tablet_bootstrap.cc:427] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:23.209197 27197 ts_tablet_manager.cc:1045] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.007s
I0619 00:12:23.210757 27197 raft_consensus.cc:325] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:23.212327 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.213733 27197 raft_consensus.cc:351] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.213949 27197 raft_consensus.cc:628] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:23.214519 27197 consensus_queue.cc:227] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:23.217368 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:23.222692 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
W0619 00:12:23.223177 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Network error: Failed to ping master at 127.24.119.60:33918: Client connection negotiation failed: client connection to 127.24.119.60:33918: connect: Connection refused (error 111)
I0619 00:12:23.223481 27196 ts_tablet_manager.cc:1073] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.000s
I0619 00:12:23.227887 27197 ts_tablet_manager.cc:1073] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.001s
I0619 00:12:23.317740 27474 rpc_server.cc:205] RPC server started. Bound to: 127.24.119.60:33918
I0619 00:12:23.319136 27474 webserver.cc:175] Starting webserver on 127.24.119.60:32921
I0619 00:12:23.319313 27474 webserver.cc:186] Document root disabled
I0619 00:12:23.327797 27474 webserver.cc:313] Webserver started. Bound to: http://127.24.119.60:32921/
I0619 00:12:23.328965 27474 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/master-2/data/info.pb
I0619 00:12:23.333633 25052 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 27474
I0619 00:12:23.337167 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 40f4b63fe7a046ea95df101ba2aed54c from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:23.337909 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet a39cff5a026248e2b668dbdde15b1d44 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:23.348546 26949 raft_consensus.cc:1081] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:23.350069 27546 consensus_queue.cc:963] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:23.361996 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet a39cff5a026248e2b668dbdde15b1d44
I0619 00:12:23.362308 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a39cff5a026248e2b668dbdde15b1d44 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.362721 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a39cff5a026248e2b668dbdde15b1d44 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.363097 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a39cff5a026248e2b668dbdde15b1d44 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.363209 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 40f4b63fe7a046ea95df101ba2aed54c
I0619 00:12:23.363318 26928 tablet_service.cc:800] Processing DeleteTablet for tablet a39cff5a026248e2b668dbdde15b1d44 with delete_type TABLET_DATA_DELETED (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.363495 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 40f4b63fe7a046ea95df101ba2aed54c on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 3a3e4803b453446b822ad9db1743779a)
I0619 00:12:23.363764 27210 tablet_replica.cc:261] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:23.363878 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 40f4b63fe7a046ea95df101ba2aed54c on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 3a3e4803b453446b822ad9db1743779a)
I0619 00:12:23.363982 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.364178 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.364248 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 40f4b63fe7a046ea95df101ba2aed54c on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 3a3e4803b453446b822ad9db1743779a)
I0619 00:12:23.364336 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.364486 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.364564 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 40f4b63fe7a046ea95df101ba2aed54c deleted and replaced by tablet 3a3e4803b453446b822ad9db1743779a
I0619 00:12:23.364704 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.364872 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.365036 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.365981 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 40f4b63fe7a046ea95df101ba2aed54c with delete_type TABLET_DATA_DELETED (replaced by tablet 3a3e4803b453446b822ad9db1743779a) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:23.366482 26533 catalog_manager.cc:4596] ReplaceTablet: tablet a39cff5a026248e2b668dbdde15b1d44 deleted and replaced by tablet 5aa9932f56a9484ead73fe7e284893bc
I0619 00:12:23.366472 27199 tablet_replica.cc:261] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:23.366732 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.366834 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 40f4b63fe7a046ea95df101ba2aed54c with delete_type TABLET_DATA_DELETED (replaced by tablet 3a3e4803b453446b822ad9db1743779a) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.366936 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.367110 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.367291 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.367507 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.367727 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.367900 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.368079 27199 raft_consensus.cc:2042] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.368340 27199 raft_consensus.cc:2069] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.368829 26791 tablet_service.cc:800] Processing DeleteTablet for tablet a39cff5a026248e2b668dbdde15b1d44 with delete_type TABLET_DATA_DELETED (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:23.369288 27200 tablet_replica.cc:261] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:23.369527 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.369767 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.369959 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.370143 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.370237 27210 raft_consensus.cc:2042] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.370354 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.370543 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.369813 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 40f4b63fe7a046ea95df101ba2aed54c with delete_type TABLET_DATA_DELETED (replaced by tablet 3a3e4803b453446b822ad9db1743779a) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:23.370761 27210 pending_rounds.cc:63] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:23.370909 27210 pending_rounds.cc:68] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391036213510144 op_type: NO_OP noop_request { }
I0619 00:12:23.371055 27063 tablet_service.cc:800] Processing DeleteTablet for tablet a39cff5a026248e2b668dbdde15b1d44 with delete_type TABLET_DATA_DELETED (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:23.371201 27210 raft_consensus.cc:2622] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:23.371412 27210 raft_consensus.cc:2069] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.371870 27199 ts_tablet_manager.cc:1369] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.374469 26533 catalog_manager.cc:3663] Got report from deleted tablet a39cff5a026248e2b668dbdde15b1d44 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc): Sending delete request for this tablet
I0619 00:12:23.374724 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.374924 27200 raft_consensus.cc:2042] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.375102 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a39cff5a026248e2b668dbdde15b1d44 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.375193 27200 raft_consensus.cc:2069] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.376372 27200 ts_tablet_manager.cc:1369] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.377023 26928 tablet_service.cc:800] Processing DeleteTablet for tablet a39cff5a026248e2b668dbdde15b1d44 with delete_type TABLET_DATA_DELETED (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.389403 27199 ts_tablet_manager.cc:1382] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.389688 27199 log.cc:986] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/40f4b63fe7a046ea95df101ba2aed54c
W0619 00:12:23.385867 27616 consensus_queue.cc:1042] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1f719e8be9c9447a9acd40728c0a5710" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I0619 00:12:23.391351 26792 tablet_service.cc:759] Processing CreateTablet for tablet 3a3e4803b453446b822ad9db1743779a (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.391561 26791 tablet_service.cc:759] Processing CreateTablet for tablet 5aa9932f56a9484ead73fe7e284893bc (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.392048 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3a3e4803b453446b822ad9db1743779a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.392843 26928 tablet_service.cc:759] Processing CreateTablet for tablet 3a3e4803b453446b822ad9db1743779a (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.393420 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3a3e4803b453446b822ad9db1743779a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.396064 26929 tablet_service.cc:759] Processing CreateTablet for tablet 5aa9932f56a9484ead73fe7e284893bc (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.396294 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5aa9932f56a9484ead73fe7e284893bc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.396625 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5aa9932f56a9484ead73fe7e284893bc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.399955 26533 catalog_manager.cc:3663] Got report from deleted tablet a39cff5a026248e2b668dbdde15b1d44 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc): Sending delete request for this tablet
I0619 00:12:23.400579 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a39cff5a026248e2b668dbdde15b1d44 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.403856 26792 ts_tablet_manager.cc:1173] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.404271 27196 ts_tablet_manager.cc:1028] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:23.404623 27196 tablet_bootstrap.cc:427] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:23.406062 27200 ts_tablet_manager.cc:1382] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.406291 27200 log.cc:986] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a39cff5a026248e2b668dbdde15b1d44
I0619 00:12:23.406963 27196 tablet_bootstrap.cc:578] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:23.412032 26928 ts_tablet_manager.cc:1173] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.412395 27197 ts_tablet_manager.cc:1028] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:23.412766 27197 tablet_bootstrap.cc:427] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:23.412962 26791 ts_tablet_manager.cc:1173] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.414844 27197 tablet_bootstrap.cc:578] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:23.395041 27063 tablet_service.cc:759] Processing CreateTablet for tablet 3a3e4803b453446b822ad9db1743779a (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:23.398540 27015 consensus_peers.cc:434] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Couldn't send request to peer 10c4e713d4c44dd8904e5f77ba815785 for tablet a39cff5a026248e2b668dbdde15b1d44. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:23.399981 27062 tablet_service.cc:759] Processing CreateTablet for tablet 5aa9932f56a9484ead73fe7e284893bc (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
W0619 00:12:23.405445 27013 consensus_peers.cc:434] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Couldn't send request to peer 1f719e8be9c9447a9acd40728c0a5710 for tablet a39cff5a026248e2b668dbdde15b1d44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:23.419354 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3a3e4803b453446b822ad9db1743779a. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.419556 27061 tablet_service.cc:800] Processing DeleteTablet for tablet a39cff5a026248e2b668dbdde15b1d44 with delete_type TABLET_DATA_DELETED (replaced by tablet 5aa9932f56a9484ead73fe7e284893bc) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:23.419788 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5aa9932f56a9484ead73fe7e284893bc. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.420174 27199 ts_tablet_manager.cc:1403] T 40f4b63fe7a046ea95df101ba2aed54c P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:23.421281 27200 ts_tablet_manager.cc:1403] T a39cff5a026248e2b668dbdde15b1d44 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:23.421437 27197 tablet_bootstrap.cc:427] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:23.421756 27197 ts_tablet_manager.cc:1045] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I0619 00:12:23.423080 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet a39cff5a026248e2b668dbdde15b1d44 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:23.423272 27200 tablet_replica.cc:261] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:23.423547 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.423791 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.424000 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.424252 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.423771 27197 raft_consensus.cc:325] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.424502 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.424453 27197 raft_consensus.cc:351] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.424682 27197 raft_consensus.cc:628] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:23.424752 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.424932 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.425130 27200 raft_consensus.cc:2042] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.425328 27197 consensus_queue.cc:227] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.426645 27210 ts_tablet_manager.cc:1369] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.427762 27197 ts_tablet_manager.cc:1073] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I0619 00:12:23.428863 27200 raft_consensus.cc:2069] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.428938 27196 tablet_bootstrap.cc:427] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:23.429639 27196 ts_tablet_manager.cc:1045] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.004s
I0619 00:12:23.430138 27199 tablet_replica.cc:261] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:23.430348 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 40f4b63fe7a046ea95df101ba2aed54c (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:23.430685 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.430922 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.431130 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.431334 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.431567 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.431814 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.432025 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(a39cff5a026248e2b668dbdde15b1d44)
I0619 00:12:23.432265 27199 raft_consensus.cc:2042] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:23.431910 27196 raft_consensus.cc:325] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.432494 27199 consensus_peers.cc:456] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:23.432560 27196 raft_consensus.cc:351] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.432719 27199 consensus_peers.cc:456] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:23.432832 27196 raft_consensus.cc:628] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:23.433185 27199 raft_consensus.cc:2069] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.434741 27196 consensus_queue.cc:227] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.436041 27199 ts_tablet_manager.cc:1369] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.434518 27200 ts_tablet_manager.cc:1369] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.436296 27210 ts_tablet_manager.cc:1382] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.436539 27210 log.cc:986] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a39cff5a026248e2b668dbdde15b1d44
I0619 00:12:23.440747 27062 ts_tablet_manager.cc:1173] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.441151 27195 ts_tablet_manager.cc:1028] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:23.441504 27195 tablet_bootstrap.cc:427] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:23.444025 27195 tablet_bootstrap.cc:578] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:23.446421 27063 ts_tablet_manager.cc:1173] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.462337 27199 ts_tablet_manager.cc:1382] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.463030 27513 raft_consensus.cc:457] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:23.463282 27513 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.718s (starting election)
I0619 00:12:23.463536 27513 raft_consensus.cc:479] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.464507 27513 leader_election.cc:243] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.464948 27513 leader_election.cc:243] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.466653 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3a3e4803b453446b822ad9db1743779a" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
W0619 00:12:23.477270 26732 leader_election.cc:306] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:23.477861 27199 log.cc:986] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a39cff5a026248e2b668dbdde15b1d44
I0619 00:12:23.483006 27612 sys_catalog.cc:188] Verifying existing consensus state
I0619 00:12:23.464754 27200 ts_tablet_manager.cc:1382] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.484367 27200 log.cc:986] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/40f4b63fe7a046ea95df101ba2aed54c
I0619 00:12:23.485074 27200 ts_tablet_manager.cc:1403] T 40f4b63fe7a046ea95df101ba2aed54c P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:23.484035 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3a3e4803b453446b822ad9db1743779a" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:23.486302 26949 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.587s (vote granted)
I0619 00:12:23.486650 26949 raft_consensus.cc:2245] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:23.488921 26732 leader_election.cc:379] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.489197 26732 leader_election.cc:270] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:23.489871 27513 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.179s (election complete)
I0619 00:12:23.490108 27513 raft_consensus.cc:2539] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:23.490309 27513 raft_consensus.cc:457] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:23.490497 27513 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.658s (starting election)
I0619 00:12:23.490716 27513 raft_consensus.cc:2784] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.491175 26929 ts_tablet_manager.cc:1173] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.492434 27197 ts_tablet_manager.cc:1028] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:23.492818 27197 tablet_bootstrap.cc:427] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:23.497813 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 40f4b63fe7a046ea95df101ba2aed54c (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:23.498608 27513 raft_consensus.cc:479] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.499583 27210 ts_tablet_manager.cc:1403] T a39cff5a026248e2b668dbdde15b1d44 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:23.500118 27196 ts_tablet_manager.cc:1073] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.070s	user 0.004s	sys 0.000s
I0619 00:12:23.500532 27196 ts_tablet_manager.cc:1028] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:23.500965 27196 tablet_bootstrap.cc:427] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:23.501837 27210 tablet_replica.cc:261] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:23.502092 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.502110 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet a39cff5a026248e2b668dbdde15b1d44 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:23.502462 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.502666 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.502848 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.503059 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.503247 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.503424 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(40f4b63fe7a046ea95df101ba2aed54c)
I0619 00:12:23.503593 27210 raft_consensus.cc:2042] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.503878 27210 raft_consensus.cc:2069] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.498656 27197 tablet_bootstrap.cc:578] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:23.508147 26949 raft_consensus.cc:1081] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:23.508667 27513 leader_election.cc:243] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.509366 27199 ts_tablet_manager.cc:1403] T a39cff5a026248e2b668dbdde15b1d44 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:23.511204 27424 consensus_queue.cc:963] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:23.512171 27196 tablet_bootstrap.cc:578] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:23.510963 27195 tablet_bootstrap.cc:427] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:23.512548 27195 ts_tablet_manager.cc:1045] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.071s	user 0.008s	sys 0.008s
I0619 00:12:23.513473 27210 ts_tablet_manager.cc:1369] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.514017 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet a39cff5a026248e2b668dbdde15b1d44 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:23.513873 27195 raft_consensus.cc:325] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.514427 27195 raft_consensus.cc:351] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.514636 27195 raft_consensus.cc:628] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:23.515100 27195 consensus_queue.cc:227] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.517347 27612 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Bootstrap starting.
I0619 00:12:23.517805 27195 ts_tablet_manager.cc:1073] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I0619 00:12:23.518684 27195 ts_tablet_manager.cc:1028] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:23.519044 27195 tablet_bootstrap.cc:427] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:23.519449 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet a39cff5a026248e2b668dbdde15b1d44 because the tablet was not found. No further retry: Not found: Tablet not found: a39cff5a026248e2b668dbdde15b1d44
I0619 00:12:23.521304 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3a3e4803b453446b822ad9db1743779a" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:23.521792 26949 raft_consensus.cc:2784] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.522030 26949 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.515s (vote granted)
I0619 00:12:23.512936 27513 leader_election.cc:243] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.530479 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3a3e4803b453446b822ad9db1743779a" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:23.531348 27472 raft_consensus.cc:457] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:23.531622 27472 raft_consensus.cc:2732] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.982s (starting election)
I0619 00:12:23.531911 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.531867 27472 raft_consensus.cc:479] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:23.532258 26732 leader_election.cc:306] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:23.532866 27472 leader_election.cc:243] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:23.533668 27472 leader_election.cc:243] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.534865 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5aa9932f56a9484ead73fe7e284893bc" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:23.530795 27195 tablet_bootstrap.cc:578] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:23.537628 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5aa9932f56a9484ead73fe7e284893bc" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:23.538373 27015 leader_election.cc:306] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:12:23.539288 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.541070 26949 raft_consensus.cc:2245] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:23.541743 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:23.542387 27013 leader_election.cc:306] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:23.542692 27013 leader_election.cc:270] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:23.543119 27472 raft_consensus.cc:2732] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.913s (election complete)
I0619 00:12:23.543335 27472 raft_consensus.cc:2487] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:23.544868 26732 leader_election.cc:379] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.545104 26732 leader_election.cc:270] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:23.545560 27513 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.736s (election complete)
I0619 00:12:23.545804 27513 raft_consensus.cc:2539] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:23.546270 27513 raft_consensus.cc:593] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:23.546733 27513 consensus_queue.cc:204] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:23.549278 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.549630 27196 tablet_bootstrap.cc:427] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:23.549957 27196 ts_tablet_manager.cc:1045] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.049s	user 0.003s	sys 0.003s
I0619 00:12:23.551215 27196 raft_consensus.cc:325] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.553634 27197 tablet_bootstrap.cc:427] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:23.554002 27197 ts_tablet_manager.cc:1045] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.061s	user 0.004s	sys 0.006s
I0619 00:12:23.555924 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.555776 27197 raft_consensus.cc:325] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.556398 27197 raft_consensus.cc:351] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.556574 27197 raft_consensus.cc:628] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:23.556816 27210 ts_tablet_manager.cc:1382] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.557020 27197 consensus_queue.cc:227] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.558997 27197 ts_tablet_manager.cc:1073] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
W0619 00:12:23.560369 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.562566 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.563601 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:23.567870 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.554765 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.569730 27196 raft_consensus.cc:351] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.571032 26533 catalog_manager.cc:3786] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:23.569927 27196 raft_consensus.cc:628] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:23.575884 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet a4028407d91d4bc09b5d967dee87110b from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:23.580806 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.557047 27210 log.cc:986] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/40f4b63fe7a046ea95df101ba2aed54c
I0619 00:12:23.584233 27210 ts_tablet_manager.cc:1403] T 40f4b63fe7a046ea95df101ba2aed54c P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:23.583971 27196 consensus_queue.cc:227] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.585217 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 40f4b63fe7a046ea95df101ba2aed54c (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
W0619 00:12:23.585695 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet a39cff5a026248e2b668dbdde15b1d44 because the tablet was not found. No further retry: Not found: Tablet not found: a39cff5a026248e2b668dbdde15b1d44
I0619 00:12:23.586246 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet a4028407d91d4bc09b5d967dee87110b from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:23.588305 27195 tablet_bootstrap.cc:427] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:23.588716 27195 ts_tablet_manager.cc:1045] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.070s	user 0.002s	sys 0.006s
I0619 00:12:23.590041 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet a4028407d91d4bc09b5d967dee87110b
I0619 00:12:23.590328 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4028407d91d4bc09b5d967dee87110b on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:23.590728 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4028407d91d4bc09b5d967dee87110b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:23.591096 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4028407d91d4bc09b5d967dee87110b on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:23.593204 26532 catalog_manager.cc:4596] ReplaceTablet: tablet a4028407d91d4bc09b5d967dee87110b deleted and replaced by tablet 7e75ff58c90a4729b884387e271fefa6
I0619 00:12:23.594344 26791 tablet_service.cc:800] Processing DeleteTablet for tablet a4028407d91d4bc09b5d967dee87110b with delete_type TABLET_DATA_DELETED (replaced by tablet 7e75ff58c90a4729b884387e271fefa6) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:23.594862 27200 tablet_replica.cc:261] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:23.594928 26929 tablet_service.cc:800] Processing DeleteTablet for tablet a4028407d91d4bc09b5d967dee87110b with delete_type TABLET_DATA_DELETED (replaced by tablet 7e75ff58c90a4729b884387e271fefa6) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.595095 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.595288 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.595371 27210 tablet_replica.cc:261] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:23.595470 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.595602 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.595651 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.595831 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.595906 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596020 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596096 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596192 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596276 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596401 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596457 27200 raft_consensus.cc:2042] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:23.596590 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596657 27200 consensus_peers.cc:456] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:23.596796 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.596873 27200 consensus_peers.cc:456] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:23.596969 27210 raft_consensus.cc:2042] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.597177 27200 pending_rounds.cc:63] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:23.597195 27210 pending_rounds.cc:63] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:23.597234 26532 catalog_manager.cc:3663] Got report from deleted tablet a4028407d91d4bc09b5d967dee87110b (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) (replaced by tablet 7e75ff58c90a4729b884387e271fefa6): Sending delete request for this tablet
I0619 00:12:23.597327 27200 pending_rounds.cc:68] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391037399273472 op_type: NO_OP noop_request { }
I0619 00:12:23.597337 27210 pending_rounds.cc:68] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391037399273472 op_type: NO_OP noop_request { }
I0619 00:12:23.597616 27210 raft_consensus.cc:2622] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:23.597882 27210 raft_consensus.cc:2069] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.597872 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a4028407d91d4bc09b5d967dee87110b on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:23.598752 26929 tablet_service.cc:800] Processing DeleteTablet for tablet a4028407d91d4bc09b5d967dee87110b with delete_type TABLET_DATA_DELETED (replaced by tablet 7e75ff58c90a4729b884387e271fefa6) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.594554 27063 tablet_service.cc:800] Processing DeleteTablet for tablet a4028407d91d4bc09b5d967dee87110b with delete_type TABLET_DATA_DELETED (replaced by tablet 7e75ff58c90a4729b884387e271fefa6) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:23.597609 27200 raft_consensus.cc:2622] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:23.602656 27200 raft_consensus.cc:2069] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.606240 27200 ts_tablet_manager.cc:1369] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.606530 26791 tablet_service.cc:759] Processing CreateTablet for tablet 7e75ff58c90a4729b884387e271fefa6 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.607164 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7e75ff58c90a4729b884387e271fefa6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.607118 26929 tablet_service.cc:759] Processing CreateTablet for tablet 7e75ff58c90a4729b884387e271fefa6 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.607708 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7e75ff58c90a4729b884387e271fefa6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.607980 27063 tablet_service.cc:759] Processing CreateTablet for tablet 7e75ff58c90a4729b884387e271fefa6 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.608552 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7e75ff58c90a4729b884387e271fefa6. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.608930 27199 tablet_replica.cc:261] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:23.609190 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.609385 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.609563 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.609761 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.610085 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.610275 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.610445 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(a4028407d91d4bc09b5d967dee87110b)
I0619 00:12:23.610616 27199 raft_consensus.cc:2042] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.610906 27199 raft_consensus.cc:2069] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.612076 27199 ts_tablet_manager.cc:1369] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
W0619 00:12:23.617754 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.619012 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.621886 26929 ts_tablet_manager.cc:1173] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.622295 27063 ts_tablet_manager.cc:1173] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.626135 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.626965 27196 ts_tablet_manager.cc:1073] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.077s	user 0.003s	sys 0.001s
W0619 00:12:23.628525 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:23.628533 27627 consensus_queue.cc:1042] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1f719e8be9c9447a9acd40728c0a5710" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
W0619 00:12:23.632251 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.632457 27197 ts_tablet_manager.cc:1028] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:23.634114 27210 ts_tablet_manager.cc:1369] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.634121 26791 ts_tablet_manager.cc:1173] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.637172 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.637426 27199 ts_tablet_manager.cc:1382] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.637734 27196 ts_tablet_manager.cc:1028] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
W0619 00:12:23.642141 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.642773 27196 tablet_bootstrap.cc:427] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:23.647935 27200 ts_tablet_manager.cc:1382] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.648186 27200 log.cc:986] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a4028407d91d4bc09b5d967dee87110b
I0619 00:12:23.648768 27196 tablet_bootstrap.cc:578] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:23.649051 27199 log.cc:986] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a4028407d91d4bc09b5d967dee87110b
I0619 00:12:23.648809 27200 ts_tablet_manager.cc:1403] T a4028407d91d4bc09b5d967dee87110b P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:23.649582 27199 ts_tablet_manager.cc:1403] T a4028407d91d4bc09b5d967dee87110b P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:23.651901 27197 tablet_bootstrap.cc:427] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:23.651837 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet a4028407d91d4bc09b5d967dee87110b (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:23.625870 27195 raft_consensus.cc:325] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.652173 27195 raft_consensus.cc:351] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.652364 27195 raft_consensus.cc:628] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:23.652858 27195 consensus_queue.cc:227] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.654316 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet a4028407d91d4bc09b5d967dee87110b (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:23.654639 27197 tablet_bootstrap.cc:578] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:23.655095 27195 ts_tablet_manager.cc:1073] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.056s	user 0.004s	sys 0.000s
I0619 00:12:23.655431 27195 ts_tablet_manager.cc:1028] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:23.655788 27195 tablet_bootstrap.cc:427] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:23.656983 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:23.662971 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.663141 27210 ts_tablet_manager.cc:1382] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.663385 27210 log.cc:986] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a4028407d91d4bc09b5d967dee87110b
I0619 00:12:23.666040 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.666326 27195 tablet_bootstrap.cc:578] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:23.670140 27195 tablet_bootstrap.cc:427] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
W0619 00:12:23.670159 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.670536 27195 ts_tablet_manager.cc:1045] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.003s
I0619 00:12:23.672399 27195 raft_consensus.cc:325] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.673068 27195 raft_consensus.cc:351] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.673276 27195 raft_consensus.cc:628] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:23.673827 27195 consensus_queue.cc:227] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.676015 27210 ts_tablet_manager.cc:1403] T a4028407d91d4bc09b5d967dee87110b P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:23.677479 27196 tablet_bootstrap.cc:427] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:23.677835 27196 ts_tablet_manager.cc:1045] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.039s	user 0.005s	sys 0.003s
I0619 00:12:23.678078 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.679927 27196 raft_consensus.cc:325] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.681740 27195 ts_tablet_manager.cc:1073] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.000s
I0619 00:12:23.680727 27196 raft_consensus.cc:351] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.684240 27196 raft_consensus.cc:628] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:23.684823 27196 consensus_queue.cc:227] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:23.687896 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.690503 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.691478 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.698058 27196 ts_tablet_manager.cc:1073] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.020s	user 0.003s	sys 0.001s
W0619 00:12:23.698918 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:23.699000 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.701006 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:23.704972 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.722551 27197 tablet_bootstrap.cc:427] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:23.722939 27197 ts_tablet_manager.cc:1045] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.071s	user 0.005s	sys 0.001s
I0619 00:12:23.724208 27197 raft_consensus.cc:325] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.724794 27197 raft_consensus.cc:351] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.724963 27197 raft_consensus.cc:628] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:23.725396 27197 consensus_queue.cc:227] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.732933 27197 ts_tablet_manager.cc:1073] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I0619 00:12:23.735862 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet a4028407d91d4bc09b5d967dee87110b (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
W0619 00:12:23.736656 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet a4028407d91d4bc09b5d967dee87110b because the tablet was not found. No further retry: Not found: Tablet not found: a4028407d91d4bc09b5d967dee87110b
I0619 00:12:23.738734 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.742449 27612 log.cc:525] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Log is configured to *not* fsync() on all Append() calls
W0619 00:12:23.743837 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.774125 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:23.779069 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.786191 27637 raft_consensus.cc:457] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:23.786479 27637 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.874s (starting election)
I0619 00:12:23.787235 27637 raft_consensus.cc:479] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.788307 27637 leader_election.cc:243] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.788758 27637 leader_election.cc:243] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:23.790004 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3a3e4803b453446b822ad9db1743779a" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:23.790611 26950 raft_consensus.cc:2175] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 10c4e713d4c44dd8904e5f77ba815785 in this term.
I0619 00:12:23.791208 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3a3e4803b453446b822ad9db1743779a" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:23.791846 26811 raft_consensus.cc:2206] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:23.793068 27015 leader_election.cc:393] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295). Message: Invalid argument: T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c for term 1 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:23.793355 27013 leader_election.cc:393] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Invalid argument: T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c in current term 1: Already voted for candidate 10c4e713d4c44dd8904e5f77ba815785 in this term.
I0619 00:12:23.793599 27013 leader_election.cc:270] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:23.794057 27637 raft_consensus.cc:2732] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.711s (election complete)
I0619 00:12:23.794284 27637 raft_consensus.cc:2784] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.802058 27637 raft_consensus.cc:2487] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:23.802727 27513 raft_consensus.cc:457] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:23.802956 27513 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.815s (starting election)
I0619 00:12:23.803197 27513 raft_consensus.cc:479] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.804185 27513 leader_election.cc:243] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.804569 27513 leader_election.cc:243] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.806491 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7e75ff58c90a4729b884387e271fefa6" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:23.806797 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7e75ff58c90a4729b884387e271fefa6" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:23.807005 27083 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.599s (vote granted)
I0619 00:12:23.807250 26950 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.769s (vote granted)
I0619 00:12:23.807242 27083 raft_consensus.cc:2245] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:23.807471 26950 raft_consensus.cc:2245] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:23.808511 26732 leader_election.cc:379] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.808768 26732 leader_election.cc:270] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:23.809478 26732 leader_election.cc:379] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.809821 27513 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.096s (election complete)
I0619 00:12:23.810067 27513 raft_consensus.cc:2539] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:23.810254 27513 raft_consensus.cc:457] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:23.810452 27513 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.856s (starting election)
I0619 00:12:23.810654 27513 raft_consensus.cc:2784] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.814591 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 5aa9932f56a9484ead73fe7e284893bc from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:23.814612 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 272d0c23372d485595f52ce73d28f56c from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:23.816941 27513 raft_consensus.cc:479] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.819278 27513 leader_election.cc:243] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.819706 27513 leader_election.cc:243] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.821450 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7e75ff58c90a4729b884387e271fefa6" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:23.821899 26950 raft_consensus.cc:2784] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.822140 26950 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.735s (vote granted)
I0619 00:12:23.823138 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7e75ff58c90a4729b884387e271fefa6" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:23.823565 27083 raft_consensus.cc:2784] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:23.823825 27083 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.185s (vote granted)
I0619 00:12:23.826408 26950 raft_consensus.cc:2245] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:23.828212 26732 leader_election.cc:379] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:23.828452 26732 leader_election.cc:270] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:23.828922 27513 raft_consensus.cc:2732] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.010s (election complete)
I0619 00:12:23.829144 27513 raft_consensus.cc:2539] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:23.829418 27513 raft_consensus.cc:593] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:23.829854 27513 consensus_queue.cc:204] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.830996 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 5aa9932f56a9484ead73fe7e284893bc
I0619 00:12:23.831202 27083 raft_consensus.cc:2245] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:23.831291 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5aa9932f56a9484ead73fe7e284893bc on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:23.831686 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5aa9932f56a9484ead73fe7e284893bc on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:23.833277 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 5aa9932f56a9484ead73fe7e284893bc with delete_type TABLET_DATA_DELETED (replaced by tablet e9045aa5906b41e2ae3e437b52f0b417) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:23.833727 27200 tablet_replica.cc:261] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:23.833963 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.834163 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.834345 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.834527 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.834760 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.834952 26732 leader_election.cc:379] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:23.835448 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 5aa9932f56a9484ead73fe7e284893bc on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:23.836181 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.836608 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 5aa9932f56a9484ead73fe7e284893bc with delete_type TABLET_DATA_DELETED (replaced by tablet e9045aa5906b41e2ae3e437b52f0b417) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.836933 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 5aa9932f56a9484ead73fe7e284893bc with delete_type TABLET_DATA_DELETED (replaced by tablet e9045aa5906b41e2ae3e437b52f0b417) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:23.837169 27210 tablet_replica.cc:261] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:23.837405 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.837604 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.837719 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 272d0c23372d485595f52ce73d28f56c
I0619 00:12:23.837812 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.837993 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.837992 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 272d0c23372d485595f52ce73d28f56c on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:23.838213 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.838405 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.838397 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 272d0c23372d485595f52ce73d28f56c on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:23.838583 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.838784 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 272d0c23372d485595f52ce73d28f56c on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:23.838784 27210 raft_consensus.cc:2042] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.839076 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 272d0c23372d485595f52ce73d28f56c deleted and replaced by tablet a98a51a19eaf4faa8b10745f2363374e
I0619 00:12:23.839049 27210 raft_consensus.cc:2069] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.839542 27199 tablet_replica.cc:261] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:23.839793 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.839988 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.840174 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.840291 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 272d0c23372d485595f52ce73d28f56c with delete_type TABLET_DATA_DELETED (replaced by tablet a98a51a19eaf4faa8b10745f2363374e) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:23.834952 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.841137 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.841078 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 272d0c23372d485595f52ce73d28f56c with delete_type TABLET_DATA_DELETED (replaced by tablet a98a51a19eaf4faa8b10745f2363374e) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:23.841322 27200 raft_consensus.cc:2042] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.841459 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 272d0c23372d485595f52ce73d28f56c with delete_type TABLET_DATA_DELETED (replaced by tablet a98a51a19eaf4faa8b10745f2363374e) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:23.841572 27200 raft_consensus.cc:2069] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.842877 27210 ts_tablet_manager.cc:1369] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.842972 27200 ts_tablet_manager.cc:1369] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.843045 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 5aa9932f56a9484ead73fe7e284893bc deleted and replaced by tablet e9045aa5906b41e2ae3e437b52f0b417
I0619 00:12:23.840359 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.847622 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.847846 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.848039 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(5aa9932f56a9484ead73fe7e284893bc)
I0619 00:12:23.848222 27199 raft_consensus.cc:2042] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.848489 27199 raft_consensus.cc:2069] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.850703 27199 ts_tablet_manager.cc:1369] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.853924 26929 tablet_service.cc:759] Processing CreateTablet for tablet e9045aa5906b41e2ae3e437b52f0b417 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.854241 26791 tablet_service.cc:759] Processing CreateTablet for tablet e9045aa5906b41e2ae3e437b52f0b417 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.854518 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9045aa5906b41e2ae3e437b52f0b417. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.854838 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9045aa5906b41e2ae3e437b52f0b417. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.856511 27063 tablet_service.cc:759] Processing CreateTablet for tablet e9045aa5906b41e2ae3e437b52f0b417 (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.857147 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9045aa5906b41e2ae3e437b52f0b417. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:23.860682 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.868399 27063 ts_tablet_manager.cc:1173] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.869546 26928 tablet_service.cc:759] Processing CreateTablet for tablet a98a51a19eaf4faa8b10745f2363374e (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.869266 26533 catalog_manager.cc:3786] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:23.870180 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a98a51a19eaf4faa8b10745f2363374e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.871567 27210 ts_tablet_manager.cc:1382] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.871834 27210 log.cc:986] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/5aa9932f56a9484ead73fe7e284893bc
I0619 00:12:23.872334 27210 ts_tablet_manager.cc:1403] T 5aa9932f56a9484ead73fe7e284893bc P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:23.872318 26792 tablet_service.cc:759] Processing CreateTablet for tablet a98a51a19eaf4faa8b10745f2363374e (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.872921 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a98a51a19eaf4faa8b10745f2363374e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.874989 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 5aa9932f56a9484ead73fe7e284893bc (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:23.878996 26928 ts_tablet_manager.cc:1173] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.879389 27197 ts_tablet_manager.cc:1028] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:23.879761 27197 tablet_bootstrap.cc:427] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:23.881568 26791 ts_tablet_manager.cc:1173] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.881985 27196 ts_tablet_manager.cc:1028] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:23.882313 27196 tablet_bootstrap.cc:427] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:23.883839 27197 tablet_bootstrap.cc:578] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:23.884337 27196 tablet_bootstrap.cc:578] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:23.889973 27210 tablet_replica.cc:261] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:23.890283 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.890506 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.890738 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.890946 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.891167 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.891371 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.891564 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.891779 27210 raft_consensus.cc:2042] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.892058 27210 raft_consensus.cc:2069] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.894927 26792 ts_tablet_manager.cc:1173] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.895974 26929 ts_tablet_manager.cc:1173] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.901209 27200 ts_tablet_manager.cc:1382] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.901451 27200 log.cc:986] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/5aa9932f56a9484ead73fe7e284893bc
I0619 00:12:23.903959 27210 ts_tablet_manager.cc:1369] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.905221 27197 tablet_bootstrap.cc:427] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:23.905555 27197 ts_tablet_manager.cc:1045] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.026s	user 0.003s	sys 0.002s
I0619 00:12:23.906889 27197 raft_consensus.cc:325] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.907508 27197 raft_consensus.cc:351] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.907726 27197 raft_consensus.cc:628] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:23.873306 27062 tablet_service.cc:759] Processing CreateTablet for tablet a98a51a19eaf4faa8b10745f2363374e (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:23.910374 27195 ts_tablet_manager.cc:1028] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:23.908139 27197 consensus_queue.cc:227] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.910604 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a98a51a19eaf4faa8b10745f2363374e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:23.910775 27195 tablet_bootstrap.cc:427] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:23.913053 27195 tablet_bootstrap.cc:578] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:23.882658 27199 ts_tablet_manager.cc:1382] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:23.913812 27199 log.cc:986] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/5aa9932f56a9484ead73fe7e284893bc
I0619 00:12:23.914292 27199 ts_tablet_manager.cc:1403] T 5aa9932f56a9484ead73fe7e284893bc P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:23.915988 27199 tablet_replica.cc:261] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:23.916226 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.916132 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 5aa9932f56a9484ead73fe7e284893bc (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:23.916422 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.916609 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.916818 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.917034 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.917235 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.917417 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.917585 27199 raft_consensus.cc:2042] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:23.919564 27199 consensus_peers.cc:456] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:23.919796 27199 consensus_peers.cc:456] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:23.920272 27199 raft_consensus.cc:2069] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.920572 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.902036 27196 tablet_bootstrap.cc:427] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:23.923722 27196 ts_tablet_manager.cc:1045] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.042s	user 0.003s	sys 0.005s
I0619 00:12:23.924362 27195 tablet_bootstrap.cc:427] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:23.924729 27195 ts_tablet_manager.cc:1045] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.002s
W0619 00:12:23.925449 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.925122 27196 raft_consensus.cc:325] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.925757 27196 raft_consensus.cc:351] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.925945 27196 raft_consensus.cc:628] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:23.926846 27199 ts_tablet_manager.cc:1369] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.926542 27196 consensus_queue.cc:227] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.931666 27195 raft_consensus.cc:325] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.932323 27195 raft_consensus.cc:351] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.932543 27195 raft_consensus.cc:628] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:23.933125 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.937463 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.938916 27196 ts_tablet_manager.cc:1073] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.000s
I0619 00:12:23.939244 27196 ts_tablet_manager.cc:1028] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:23.939611 27196 tablet_bootstrap.cc:427] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:23.933123 27195 consensus_queue.cc:227] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:23.940280 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.944546 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:23.945106 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.945432 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.946066 27197 ts_tablet_manager.cc:1073] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.040s	user 0.004s	sys 0.000s
I0619 00:12:23.946328 27062 ts_tablet_manager.cc:1173] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:23.946496 27197 ts_tablet_manager.cc:1028] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
W0619 00:12:23.949445 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.949605 27196 tablet_bootstrap.cc:578] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:23.951484 27197 tablet_bootstrap.cc:427] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:23.952569 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:23.952641 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.953202 27200 ts_tablet_manager.cc:1403] T 5aa9932f56a9484ead73fe7e284893bc P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:23.954968 27200 tablet_replica.cc:261] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:23.955150 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 5aa9932f56a9484ead73fe7e284893bc (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:23.956475 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(272d0c23372d485595f52ce73d28f56c)
W0619 00:12:23.956539 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.953677 27195 ts_tablet_manager.cc:1073] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.029s	user 0.003s	sys 0.000s
I0619 00:12:23.957156 27195 ts_tablet_manager.cc:1028] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:23.957267 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.957465 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.957563 27195 tablet_bootstrap.cc:427] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:23.957689 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.957731 27197 tablet_bootstrap.cc:578] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:23.957904 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.958096 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.958287 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(272d0c23372d485595f52ce73d28f56c)
I0619 00:12:23.958467 27200 raft_consensus.cc:2042] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:23.958750 27200 raft_consensus.cc:2069] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:23.963071 27210 ts_tablet_manager.cc:1382] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.964952 27195 tablet_bootstrap.cc:578] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:23.970722 27210 log.cc:986] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/272d0c23372d485595f52ce73d28f56c
I0619 00:12:23.971351 27200 ts_tablet_manager.cc:1369] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:23.971372 27210 ts_tablet_manager.cc:1403] T 272d0c23372d485595f52ce73d28f56c P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:23.973322 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 272d0c23372d485595f52ce73d28f56c (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:23.972815 27195 tablet_bootstrap.cc:427] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:23.973881 27197 tablet_bootstrap.cc:427] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:23.974002 27195 ts_tablet_manager.cc:1045] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.001s
I0619 00:12:23.974272 27197 ts_tablet_manager.cc:1045] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.001s
I0619 00:12:23.974460 27196 tablet_bootstrap.cc:427] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:23.974830 27196 ts_tablet_manager.cc:1045] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.035s	user 0.006s	sys 0.000s
I0619 00:12:23.975312 27195 raft_consensus.cc:325] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.975952 27195 raft_consensus.cc:351] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.976140 27195 raft_consensus.cc:628] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:23.975803 27197 raft_consensus.cc:325] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.976573 27197 raft_consensus.cc:351] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.976189 27196 raft_consensus.cc:325] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.976804 27196 raft_consensus.cc:351] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:23.976826 27197 raft_consensus.cc:628] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:23.976989 27196 raft_consensus.cc:628] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:23.976683 27195 consensus_queue.cc:227] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.977414 27196 consensus_queue.cc:227] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.977537 27197 consensus_queue.cc:227] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:23.977290 27199 ts_tablet_manager.cc:1382] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:23.978643 27199 log.cc:986] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/272d0c23372d485595f52ce73d28f56c
I0619 00:12:23.981402 27199 ts_tablet_manager.cc:1403] T 272d0c23372d485595f52ce73d28f56c P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:23.986398 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.986944 26950 raft_consensus.cc:1081] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:23.990209 27513 consensus_queue.cc:963] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:23.991050 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.992749 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:23.993341 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 272d0c23372d485595f52ce73d28f56c (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:23.995609 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:23.990428 27196 ts_tablet_manager.cc:1073] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.001s
W0619 00:12:24.000501 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.000537 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.004035 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.007351 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.009438 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.011734 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.013074 27083 raft_consensus.cc:1081] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:24.016271 27424 consensus_queue.cc:963] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:24.022713 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.023963 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.025777 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.027637 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.029836 27195 ts_tablet_manager.cc:1073] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.056s	user 0.003s	sys 0.002s
I0619 00:12:24.030931 27197 ts_tablet_manager.cc:1073] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.056s	user 0.004s	sys 0.000s
I0619 00:12:24.032282 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.037303 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.040794 27200 ts_tablet_manager.cc:1382] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.040997 27200 log.cc:986] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/272d0c23372d485595f52ce73d28f56c
I0619 00:12:24.041517 27200 ts_tablet_manager.cc:1403] T 272d0c23372d485595f52ce73d28f56c P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:24.043249 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 272d0c23372d485595f52ce73d28f56c (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:24.049077 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.049628 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet a98a51a19eaf4faa8b10745f2363374e from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:24.051368 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.051399 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.051918 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 3a3e4803b453446b822ad9db1743779a from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:24.057744 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.058864 27424 raft_consensus.cc:457] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.059100 27424 raft_consensus.cc:2732] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.503s (starting election)
I0619 00:12:24.059351 27424 raft_consensus.cc:479] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.060319 27424 leader_election.cc:243] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.060729 27424 leader_election.cc:243] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.060909 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet a98a51a19eaf4faa8b10745f2363374e
I0619 00:12:24.061899 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a98a51a19eaf4faa8b10745f2363374e on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.062573 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a98a51a19eaf4faa8b10745f2363374e on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.063282 26929 tablet_service.cc:800] Processing DeleteTablet for tablet a98a51a19eaf4faa8b10745f2363374e with delete_type TABLET_DATA_DELETED (replaced by tablet 02d950f63c9c46d381be1c3b31f5a2e4) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.063377 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.064049 26792 tablet_service.cc:800] Processing DeleteTablet for tablet a98a51a19eaf4faa8b10745f2363374e with delete_type TABLET_DATA_DELETED (replaced by tablet 02d950f63c9c46d381be1c3b31f5a2e4) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:24.064981 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a98a51a19eaf4faa8b10745f2363374e on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.065045 27081 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a98a51a19eaf4faa8b10745f2363374e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:24.063259 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a98a51a19eaf4faa8b10745f2363374e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:24.065523 27081 raft_consensus.cc:2732] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.513s (vote granted)
I0619 00:12:24.065557 26533 catalog_manager.cc:4596] ReplaceTablet: tablet a98a51a19eaf4faa8b10745f2363374e deleted and replaced by tablet 02d950f63c9c46d381be1c3b31f5a2e4
I0619 00:12:24.065677 26950 raft_consensus.cc:2732] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.642s (vote granted)
I0619 00:12:24.065764 27081 raft_consensus.cc:2245] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:24.065896 26950 raft_consensus.cc:2245] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:24.066663 27062 tablet_service.cc:800] Processing DeleteTablet for tablet a98a51a19eaf4faa8b10745f2363374e with delete_type TABLET_DATA_DELETED (replaced by tablet 02d950f63c9c46d381be1c3b31f5a2e4) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:24.067155 27199 tablet_replica.cc:261] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:24.067395 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.067597 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.067806 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.067988 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.068022 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 3a3e4803b453446b822ad9db1743779a
I0619 00:12:24.068205 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.068289 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3a3e4803b453446b822ad9db1743779a on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.068401 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.068583 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.068671 27200 tablet_replica.cc:261] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:24.068780 27199 raft_consensus.cc:2042] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.068910 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.069038 27199 raft_consensus.cc:2069] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.069104 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.069231 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 3a3e4803b453446b822ad9db1743779a with delete_type TABLET_DATA_DELETED (replaced by tablet c000367cc97d42188f684e5f5df4f9f3) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.070008 26732 leader_election.cc:379] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.070058 26533 catalog_manager.cc:3663] Got report from deleted tablet 3a3e4803b453446b822ad9db1743779a (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet c000367cc97d42188f684e5f5df4f9f3): Sending delete request for this tablet
I0619 00:12:24.070279 26732 leader_election.cc:270] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:24.070770 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3a3e4803b453446b822ad9db1743779a on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.070869 26732 leader_election.cc:379] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.070587 27199 ts_tablet_manager.cc:1369] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.071725 27513 raft_consensus.cc:2732] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.853s (election complete)
I0619 00:12:24.071951 27513 raft_consensus.cc:2539] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:24.071746 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 3a3e4803b453446b822ad9db1743779a with delete_type TABLET_DATA_DELETED (replaced by tablet c000367cc97d42188f684e5f5df4f9f3) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:24.072149 27513 raft_consensus.cc:457] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:24.072346 27513 raft_consensus.cc:2732] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.540s (starting election)
I0619 00:12:24.072548 27513 raft_consensus.cc:2784] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.073199 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3a3e4803b453446b822ad9db1743779a on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.076086 27210 tablet_replica.cc:261] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:24.076345 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.076577 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.076812 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.077021 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.076989 26792 tablet_service.cc:759] Processing CreateTablet for tablet 02d950f63c9c46d381be1c3b31f5a2e4 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.077302 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.077527 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.077592 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 02d950f63c9c46d381be1c3b31f5a2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.077793 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.078001 27210 raft_consensus.cc:2042] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.078302 27210 raft_consensus.cc:2069] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.079741 27210 ts_tablet_manager.cc:1369] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.080626 27062 tablet_service.cc:759] Processing CreateTablet for tablet 02d950f63c9c46d381be1c3b31f5a2e4 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.081207 27062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 02d950f63c9c46d381be1c3b31f5a2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.081084 27513 raft_consensus.cc:479] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.082072 27513 leader_election.cc:243] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.082482 27513 leader_election.cc:243] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.080747 26929 tablet_service.cc:759] Processing CreateTablet for tablet 02d950f63c9c46d381be1c3b31f5a2e4 (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.085067 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 02d950f63c9c46d381be1c3b31f5a2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.069290 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.086199 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.086414 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.086673 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.086856 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(a98a51a19eaf4faa8b10745f2363374e)
I0619 00:12:24.087025 27200 raft_consensus.cc:2042] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.084259 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a98a51a19eaf4faa8b10745f2363374e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:24.088539 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a98a51a19eaf4faa8b10745f2363374e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:24.088996 27200 raft_consensus.cc:2069] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.090379 27200 ts_tablet_manager.cc:1369] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.090350 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 3a3e4803b453446b822ad9db1743779a with delete_type TABLET_DATA_DELETED (replaced by tablet c000367cc97d42188f684e5f5df4f9f3) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:24.090868 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3a3e4803b453446b822ad9db1743779a on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.091779 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 3a3e4803b453446b822ad9db1743779a with delete_type TABLET_DATA_DELETED (replaced by tablet c000367cc97d42188f684e5f5df4f9f3) from {username='jenkins'} at 127.24.119.3:56160
W0619 00:12:24.093118 26732 leader_election.cc:306] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: Tablet not RUNNING: STOPPED
W0619 00:12:24.093634 26732 leader_election.cc:306] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:24.093873 26732 leader_election.cc:270] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
W0619 00:12:24.094241 26732 raft_consensus.cc:2442] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Unable to run election callback: Service unavailable: Thread pool token was shut down
I0619 00:12:24.097514 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 3a3e4803b453446b822ad9db1743779a deleted and replaced by tablet c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.098595 27062 ts_tablet_manager.cc:1173] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.101076 26929 ts_tablet_manager.cc:1173] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.101233 27195 ts_tablet_manager.cc:1028] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:24.101476 27197 ts_tablet_manager.cc:1028] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:24.101708 27195 tablet_bootstrap.cc:427] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:24.101855 27197 tablet_bootstrap.cc:427] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:24.102708 26791 tablet_service.cc:759] Processing CreateTablet for tablet c000367cc97d42188f684e5f5df4f9f3 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.103302 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c000367cc97d42188f684e5f5df4f9f3. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:24.103487 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.105201 27195 tablet_bootstrap.cc:578] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:24.106963 26928 tablet_service.cc:759] Processing CreateTablet for tablet c000367cc97d42188f684e5f5df4f9f3 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.107547 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c000367cc97d42188f684e5f5df4f9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.110000 27063 tablet_service.cc:759] Processing CreateTablet for tablet c000367cc97d42188f684e5f5df4f9f3 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.110615 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c000367cc97d42188f684e5f5df4f9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.112731 26791 ts_tablet_manager.cc:1173] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.113087 27196 ts_tablet_manager.cc:1028] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:24.113420 27196 tablet_bootstrap.cc:427] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:24.115535 27196 tablet_bootstrap.cc:578] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:24.111055 27199 ts_tablet_manager.cc:1382] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.117592 27199 log.cc:986] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/a98a51a19eaf4faa8b10745f2363374e
I0619 00:12:24.118748 26792 ts_tablet_manager.cc:1173] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.119071 27199 ts_tablet_manager.cc:1403] T a98a51a19eaf4faa8b10745f2363374e P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:24.122438 27210 ts_tablet_manager.cc:1382] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.122506 26928 ts_tablet_manager.cc:1173] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.122757 27210 log.cc:986] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/a98a51a19eaf4faa8b10745f2363374e
I0619 00:12:24.123759 27210 ts_tablet_manager.cc:1403] T a98a51a19eaf4faa8b10745f2363374e P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:24.125159 27063 ts_tablet_manager.cc:1173] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.125641 27195 tablet_bootstrap.cc:427] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:24.126240 27195 ts_tablet_manager.cc:1045] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.025s	user 0.006s	sys 0.000s
I0619 00:12:24.127705 27195 raft_consensus.cc:325] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.129556 27196 tablet_bootstrap.cc:427] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:24.129953 27196 ts_tablet_manager.cc:1045] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.003s
I0619 00:12:24.130034 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet a98a51a19eaf4faa8b10745f2363374e (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:24.130471 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet a98a51a19eaf4faa8b10745f2363374e (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:24.128270 27637 raft_consensus.cc:457] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.130945 27637 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.021s (starting election)
I0619 00:12:24.131229 27210 tablet_replica.cc:261] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:24.131451 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.131659 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.131080 27199 tablet_replica.cc:261] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:24.131234 27637 raft_consensus.cc:479] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.131834 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.132210 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.132272 27637 leader_election.cc:243] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.132431 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.132650 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.132704 27637 leader_election.cc:243] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.132843 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.132426 27196 raft_consensus.cc:325] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.133455 27196 raft_consensus.cc:351] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.133675 27196 raft_consensus.cc:628] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:24.134173 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.134217 27196 consensus_queue.cc:227] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.134799 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.134752 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9045aa5906b41e2ae3e437b52f0b417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:24.135476 26811 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.727s (vote granted)
I0619 00:12:24.135339 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9045aa5906b41e2ae3e437b52f0b417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:24.135723 26811 raft_consensus.cc:2245] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.135828 26950 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.129s (vote granted)
I0619 00:12:24.135484 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.136046 26950 raft_consensus.cc:2245] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.136747 27015 leader_election.cc:379] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.136994 27015 leader_election.cc:270] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:24.137562 27637 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.179s (election complete)
I0619 00:12:24.137820 27637 raft_consensus.cc:2539] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:24.138031 27637 raft_consensus.cc:457] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:24.137995 27013 leader_election.cc:379] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.138447 27637 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.144s (starting election)
I0619 00:12:24.130913 27195 raft_consensus.cc:351] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.138708 27637 raft_consensus.cc:2784] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.138901 27195 raft_consensus.cc:628] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:24.139596 27195 consensus_queue.cc:227] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.144033 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.133010 27210 raft_consensus.cc:2042] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.144234 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.144508 27210 raft_consensus.cc:2069] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.144961 27195 ts_tablet_manager.cc:1073] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.001s
I0619 00:12:24.146700 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.148767 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.145897 27637 raft_consensus.cc:479] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.147296 27195 ts_tablet_manager.cc:1028] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:24.150218 27195 tablet_bootstrap.cc:427] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:24.150770 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.150909 27637 leader_election.cc:243] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.153262 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9045aa5906b41e2ae3e437b52f0b417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
W0619 00:12:24.153705 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.153728 26811 raft_consensus.cc:2784] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.153961 26811 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.735s (vote granted)
I0619 00:12:24.155352 27210 ts_tablet_manager.cc:1369] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.155973 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.156033 27200 ts_tablet_manager.cc:1382] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.156268 27200 log.cc:986] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/a98a51a19eaf4faa8b10745f2363374e
I0619 00:12:24.156816 27200 ts_tablet_manager.cc:1403] T a98a51a19eaf4faa8b10745f2363374e P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:24.156728 27513 raft_consensus.cc:457] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.157747 27513 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.103s (starting election)
I0619 00:12:24.155098 27637 leader_election.cc:243] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.157989 27513 raft_consensus.cc:479] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.158890 26811 raft_consensus.cc:2245] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:24.158936 27513 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.159453 27513 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.159871 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9045aa5906b41e2ae3e437b52f0b417" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:24.160105 27015 leader_election.cc:379] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.160336 27015 leader_election.cc:270] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:24.160334 26950 raft_consensus.cc:2784] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.160560 26950 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.868s (vote granted)
W0619 00:12:24.161176 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.161557 27637 raft_consensus.cc:2732] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.984s (election complete)
I0619 00:12:24.161926 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:24.161988 27637 raft_consensus.cc:2539] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:24.162389 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.164427 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:24.164598 26732 leader_election.cc:306] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:24.163223 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.165948 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet a98a51a19eaf4faa8b10745f2363374e (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:24.166520 27200 tablet_replica.cc:261] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:24.166635 26950 raft_consensus.cc:2245] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:24.167244 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.167431 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.167651 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.167625 27013 leader_election.cc:379] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.167829 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.168006 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.168220 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.168416 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.168613 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(3a3e4803b453446b822ad9db1743779a)
I0619 00:12:24.168788 27200 raft_consensus.cc:2042] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:24.168982 27200 consensus_peers.cc:456] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:24.169169 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:24.169610 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.170317 27195 tablet_bootstrap.cc:578] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:24.165897 27199 raft_consensus.cc:2042] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.170509 27637 raft_consensus.cc:593] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:24.171573 27199 raft_consensus.cc:2069] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.174692 27200 consensus_peers.cc:456] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:24.175169 27200 raft_consensus.cc:2069] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.173140 27637 consensus_queue.cc:204] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:24.166712 26732 leader_election.cc:306] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:24.176762 26732 leader_election.cc:270] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:24.177199 27513 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.585s (election complete)
W0619 00:12:24.172549 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.173303 27196 ts_tablet_manager.cc:1073] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.043s	user 0.003s	sys 0.000s
I0619 00:12:24.177425 27513 raft_consensus.cc:2487] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:24.177796 27196 ts_tablet_manager.cc:1028] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:24.178367 27197 tablet_bootstrap.cc:578] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:24.178161 27196 tablet_bootstrap.cc:427] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:24.181772 27200 ts_tablet_manager.cc:1369] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.183239 27199 ts_tablet_manager.cc:1369] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.184958 27197 tablet_bootstrap.cc:427] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:24.185281 27197 ts_tablet_manager.cc:1045] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.084s	user 0.005s	sys 0.002s
I0619 00:12:24.187111 27210 ts_tablet_manager.cc:1382] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.189242 27196 tablet_bootstrap.cc:578] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:24.186763 27197 raft_consensus.cc:325] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.193083 27197 raft_consensus.cc:351] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.193009 27210 log.cc:986] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/3a3e4803b453446b822ad9db1743779a
I0619 00:12:24.193445 27197 raft_consensus.cc:628] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:24.194427 27195 tablet_bootstrap.cc:427] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:24.194017 27197 consensus_queue.cc:227] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.198334 27197 ts_tablet_manager.cc:1073] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.000s
I0619 00:12:24.198720 27197 ts_tablet_manager.cc:1028] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:24.198655 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.201416 27197 tablet_bootstrap.cc:427] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:24.201745 26532 catalog_manager.cc:3786] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
W0619 00:12:24.204723 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.205929 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.206216 27195 ts_tablet_manager.cc:1045] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.056s	user 0.007s	sys 0.002s
I0619 00:12:24.207350 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.211536 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.211906 27195 raft_consensus.cc:325] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.212535 27195 raft_consensus.cc:351] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.212764 27195 raft_consensus.cc:628] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:24.213372 27210 ts_tablet_manager.cc:1403] T 3a3e4803b453446b822ad9db1743779a P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:24.213704 27195 consensus_queue.cc:227] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.215965 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 3a3e4803b453446b822ad9db1743779a (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.216732 27197 tablet_bootstrap.cc:578] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:24.218444 27195 ts_tablet_manager.cc:1073] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.001s
I0619 00:12:24.219235 27083 raft_consensus.cc:1081] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:24.220645 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.225334 27513 consensus_queue.cc:963] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:24.227452 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.233420 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.236907 27200 ts_tablet_manager.cc:1382] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.237145 27200 log.cc:986] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/3a3e4803b453446b822ad9db1743779a
I0619 00:12:24.237788 27200 ts_tablet_manager.cc:1403] T 3a3e4803b453446b822ad9db1743779a P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
W0619 00:12:24.242693 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.243885 27637 raft_consensus.cc:457] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.244043 27197 tablet_bootstrap.cc:427] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:24.244143 27637 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.064s (starting election)
I0619 00:12:24.244346 27197 ts_tablet_manager.cc:1045] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.045s	user 0.003s	sys 0.005s
I0619 00:12:24.244385 27637 raft_consensus.cc:479] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.245353 27637 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.245740 27637 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.245693 27197 raft_consensus.cc:325] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.246289 27197 raft_consensus.cc:351] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.248354 26950 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:24.248558 27197 raft_consensus.cc:628] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:24.247642 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:24.251754 26811 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.003s (vote granted)
I0619 00:12:24.251974 26811 raft_consensus.cc:2245] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.253033 27015 leader_election.cc:379] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.253249 27015 leader_election.cc:270] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:24.253757 27637 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.820s (election complete)
I0619 00:12:24.253969 27637 raft_consensus.cc:2539] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:24.254156 27637 raft_consensus.cc:457] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:24.254338 27637 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.086s (starting election)
I0619 00:12:24.254526 27637 raft_consensus.cc:2784] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.258607 26949 raft_consensus.cc:1081] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:24.259349 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 3a3e4803b453446b822ad9db1743779a (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.259668 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.261807 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.262059 27637 raft_consensus.cc:479] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.262938 27424 consensus_queue.cc:963] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:24.264921 27637 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.265802 27637 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.266510 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:24.266986 26811 raft_consensus.cc:2784] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.267217 26811 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.507s (vote granted)
I0619 00:12:24.267349 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:24.268021 27199 ts_tablet_manager.cc:1382] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W0619 00:12:24.268265 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.269384 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.248364 27196 tablet_bootstrap.cc:427] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:24.273129 27196 ts_tablet_manager.cc:1045] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.095s	user 0.005s	sys 0.006s
I0619 00:12:24.274371 26811 raft_consensus.cc:2245] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:24.275718 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.275816 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet c000367cc97d42188f684e5f5df4f9f3 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:24.276800 27015 leader_election.cc:379] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.277060 27015 leader_election.cc:270] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:24.268366 27199 log.cc:986] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/3a3e4803b453446b822ad9db1743779a
I0619 00:12:24.277544 27637 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.855s (election complete)
I0619 00:12:24.277806 27637 raft_consensus.cc:2539] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:24.279373 27637 raft_consensus.cc:593] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:24.279865 27199 ts_tablet_manager.cc:1403] T 3a3e4803b453446b822ad9db1743779a P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:24.280586 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.278147 27196 raft_consensus.cc:325] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.280951 27196 raft_consensus.cc:351] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.281144 27196 raft_consensus.cc:628] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:24.283071 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 3a3e4803b453446b822ad9db1743779a (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.284474 27196 consensus_queue.cc:227] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:24.285291 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet 3a3e4803b453446b822ad9db1743779a because the tablet was not found. No further retry: Not found: Tablet not found: 3a3e4803b453446b822ad9db1743779a
I0619 00:12:24.293820 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.279830 27637 consensus_queue.cc:204] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.298692 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.299314 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.300575 26791 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:24.301089 27200 tablet_replica.cc:261] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:24.301302 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.301479 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.301662 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.301820 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.302006 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.302170 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.302323 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.302475 27200 raft_consensus.cc:2042] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.302727 27200 raft_consensus.cc:2069] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.304224 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.305677 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.306143 26928 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.2:46317
W0619 00:12:24.312156 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.317227 27197 consensus_queue.cc:227] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.318187 26950 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.636s (vote granted)
I0619 00:12:24.318423 26950 raft_consensus.cc:2245] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.318814 27200 ts_tablet_manager.cc:1369] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.319360 26949 raft_consensus.cc:2220] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:12:24.320750 27328 raft_consensus.cc:457] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.320971 27328 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.973s (starting election)
I0619 00:12:24.321198 27328 raft_consensus.cc:479] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:24.308037 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet c000367cc97d42188f684e5f5df4f9f3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c000367cc97d42188f684e5f5df4f9f3 already in progress: creating tablet
I0619 00:12:24.322691 26521 catalog_manager.cc:2891] Scheduling retry of c000367cc97d42188f684e5f5df4f9f3 Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 65 ms (attempt = 1)
I0619 00:12:24.322842 27328 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.323240 27328 leader_election.cc:243] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.323429 26532 catalog_manager.cc:4596] ReplaceTablet: tablet c000367cc97d42188f684e5f5df4f9f3 deleted and replaced by tablet bf54a7ce52744b50ac35376a9cda49fa
I0619 00:12:24.323998 27063 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:24.311014 26533 catalog_manager.cc:3663] Got report from deleted tablet c000367cc97d42188f684e5f5df4f9f3 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa): Sending delete request for this tablet
I0619 00:12:24.324484 27199 tablet_replica.cc:261] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:24.324746 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.324749 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.324946 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.325130 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.325314 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.325522 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.325731 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.325917 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.326088 27199 raft_consensus.cc:2042] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:24.326282 27199 consensus_peers.cc:456] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:24.326478 27199 consensus_peers.cc:456] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:24.324172 27013 leader_election.cc:379] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.326884 27199 pending_rounds.cc:63] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:24.327031 27199 pending_rounds.cc:68] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391042234220544 op_type: NO_OP noop_request { }
I0619 00:12:24.327317 27199 raft_consensus.cc:2622] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:24.327391 27013 leader_election.cc:393] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote denied by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248). Message: Service unavailable: T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate a2c1dfdaa6d442539e4342d184d2167c for term 1 because replica is already servicing an update from a current leader or another vote.
I0619 00:12:24.328730 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:24.328971 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c000367cc97d42188f684e5f5df4f9f3" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:24.329181 27063 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:24.329509 27199 raft_consensus.cc:2069] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:24.330602 26876 leader_election.cc:306] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: Tablet not RUNNING: STOPPED
W0619 00:12:24.330886 26874 leader_election.cc:306] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:24.331146 26874 leader_election.cc:270] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:24.331600 27328 raft_consensus.cc:2732] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.912s (election complete)
I0619 00:12:24.331830 27328 raft_consensus.cc:2487] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:24.333438 27199 ts_tablet_manager.cc:1369] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.336655 26791 tablet_service.cc:759] Processing CreateTablet for tablet bf54a7ce52744b50ac35376a9cda49fa (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.337033 27063 tablet_service.cc:759] Processing CreateTablet for tablet bf54a7ce52744b50ac35376a9cda49fa (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.337275 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bf54a7ce52744b50ac35376a9cda49fa. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.337162 26928 tablet_service.cc:759] Processing CreateTablet for tablet bf54a7ce52744b50ac35376a9cda49fa (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.337615 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bf54a7ce52744b50ac35376a9cda49fa. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.337846 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bf54a7ce52744b50ac35376a9cda49fa. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:24.345854 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.348697 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.349236 26791 ts_tablet_manager.cc:1173] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.352766 26533 catalog_manager.cc:3663] Got report from deleted tablet c000367cc97d42188f684e5f5df4f9f3 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa): Sending delete request for this tablet
W0619 00:12:24.353144 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.353576 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.355561 26929 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.355641 27196 ts_tablet_manager.cc:1073] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.082s	user 0.004s	sys 0.003s
I0619 00:12:24.356000 27196 ts_tablet_manager.cc:1028] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:24.356329 27196 tablet_bootstrap.cc:427] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:24.360358 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 7e75ff58c90a4729b884387e271fefa6 from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:24.361232 26521 catalog_manager.cc:3083] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet c000367cc97d42188f684e5f5df4f9f3 with error code TABLET_NOT_RUNNING: Already present: State transition of tablet c000367cc97d42188f684e5f5df4f9f3 already in progress: creating tablet
I0619 00:12:24.361508 26521 catalog_manager.cc:2891] Scheduling retry of c000367cc97d42188f684e5f5df4f9f3 Delete Tablet RPC for TS=1f719e8be9c9447a9acd40728c0a5710 with a delay of 29 ms (attempt = 1)
I0619 00:12:24.366641 27196 tablet_bootstrap.cc:578] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:24.372368 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.375705 27063 ts_tablet_manager.cc:1173] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.375891 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 7e75ff58c90a4729b884387e271fefa6
I0619 00:12:24.376068 27195 ts_tablet_manager.cc:1028] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:24.376211 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7e75ff58c90a4729b884387e271fefa6 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:24.376412 27195 tablet_bootstrap.cc:427] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:24.376624 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7e75ff58c90a4729b884387e271fefa6 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 2e2867d8ea324cddb373bcfebb206143)
W0619 00:12:24.377082 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.377616 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7e75ff58c90a4729b884387e271fefa6 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:24.377939 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 7e75ff58c90a4729b884387e271fefa6 deleted and replaced by tablet 2e2867d8ea324cddb373bcfebb206143
I0619 00:12:24.378533 27195 tablet_bootstrap.cc:578] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:24.378274 27197 ts_tablet_manager.cc:1073] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.134s	user 0.005s	sys 0.002s
W0619 00:12:24.364048 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.380707 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 7e75ff58c90a4729b884387e271fefa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e2867d8ea324cddb373bcfebb206143) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.380743 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 7e75ff58c90a4729b884387e271fefa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e2867d8ea324cddb373bcfebb206143) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:24.381181 27210 tablet_replica.cc:261] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:24.381400 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.381589 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.381791 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.381971 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.382180 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.382362 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.382544 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.382735 27210 raft_consensus.cc:2042] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.382995 27210 raft_consensus.cc:2069] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.383028 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.384513 26928 ts_tablet_manager.cc:1173] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:24.387588 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.387910 27197 ts_tablet_manager.cc:1028] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:24.388281 27197 tablet_bootstrap.cc:427] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:24.388765 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.390223 26928 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.391355 26521 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.392289 26928 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.392570 27199 ts_tablet_manager.cc:1382] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.392846 27199 log.cc:986] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.391966 27210 ts_tablet_manager.cc:1369] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.394338 26928 tablet_service.cc:759] Processing CreateTablet for tablet 2e2867d8ea324cddb373bcfebb206143 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.394568 26792 tablet_service.cc:759] Processing CreateTablet for tablet 2e2867d8ea324cddb373bcfebb206143 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.394605 27063 tablet_service.cc:759] Processing CreateTablet for tablet 2e2867d8ea324cddb373bcfebb206143 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.394990 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2e2867d8ea324cddb373bcfebb206143. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.395211 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2e2867d8ea324cddb373bcfebb206143. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.395238 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2e2867d8ea324cddb373bcfebb206143. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.397475 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 7e75ff58c90a4729b884387e271fefa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e2867d8ea324cddb373bcfebb206143) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:24.385656 26533 catalog_manager.cc:3663] Got report from deleted tablet c000367cc97d42188f684e5f5df4f9f3 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa): Sending delete request for this tablet
I0619 00:12:24.398305 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c000367cc97d42188f684e5f5df4f9f3 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.399257 26929 tablet_service.cc:800] Processing DeleteTablet for tablet c000367cc97d42188f684e5f5df4f9f3 with delete_type TABLET_DATA_DELETED (replaced by tablet bf54a7ce52744b50ac35376a9cda49fa) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.401315 27195 tablet_bootstrap.cc:427] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:24.402076 27195 ts_tablet_manager.cc:1045] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.026s	user 0.004s	sys 0.004s
I0619 00:12:24.404129 27195 raft_consensus.cc:325] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.405113 27195 raft_consensus.cc:351] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.405412 27195 raft_consensus.cc:628] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:24.406213 27195 consensus_queue.cc:227] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.410337 27195 ts_tablet_manager.cc:1073] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I0619 00:12:24.414712 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.418773 27637 raft_consensus.cc:457] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.418989 27637 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.530s (starting election)
I0619 00:12:24.419214 27637 raft_consensus.cc:479] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.420151 27637 leader_election.cc:243] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.420519 27637 leader_election.cc:243] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.422049 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "02d950f63c9c46d381be1c3b31f5a2e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
W0619 00:12:24.422417 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.422544 26948 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.728s (vote granted)
I0619 00:12:24.422845 26948 raft_consensus.cc:2245] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.423840 27013 leader_election.cc:379] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.424089 27013 leader_election.cc:270] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:24.424329 27197 tablet_bootstrap.cc:578] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:24.424649 27637 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.582s (election complete)
I0619 00:12:24.424862 27637 raft_consensus.cc:2539] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:24.425055 27637 raft_consensus.cc:457] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:24.425248 27637 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.800s (starting election)
I0619 00:12:24.425284 27200 ts_tablet_manager.cc:1382] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.425417 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.425442 27637 raft_consensus.cc:2784] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.425545 27200 log.cc:986] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.417057 26792 ts_tablet_manager.cc:1173] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.427860 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "02d950f63c9c46d381be1c3b31f5a2e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:24.428303 26811 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.170s (vote granted)
I0619 00:12:24.428521 26811 raft_consensus.cc:2245] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.428786 27199 ts_tablet_manager.cc:1403] T c000367cc97d42188f684e5f5df4f9f3 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:24.429929 27015 leader_election.cc:379] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.432448 27210 ts_tablet_manager.cc:1382] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.432706 27210 log.cc:986] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/7e75ff58c90a4729b884387e271fefa6
I0619 00:12:24.433223 27210 ts_tablet_manager.cc:1403] T 7e75ff58c90a4729b884387e271fefa6 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:24.435964 27196 tablet_bootstrap.cc:427] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:24.436321 27196 ts_tablet_manager.cc:1045] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.080s	user 0.007s	sys 0.001s
I0619 00:12:24.437690 27196 raft_consensus.cc:325] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.438453 27196 raft_consensus.cc:351] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.438791 27196 raft_consensus.cc:628] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:24.439160 27200 ts_tablet_manager.cc:1403] T c000367cc97d42188f684e5f5df4f9f3 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:24.439903 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet c000367cc97d42188f684e5f5df4f9f3 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.440255 27199 tablet_replica.cc:261] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
W0619 00:12:24.440440 26521 catalog_manager.cc:3071] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): delete failed for tablet c000367cc97d42188f684e5f5df4f9f3 because the tablet was not found. No further retry: Not found: Tablet not found: c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.440507 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.440755 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.440970 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.441179 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.441416 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.441629 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.441867 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.442073 27199 raft_consensus.cc:2042] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.442370 27199 raft_consensus.cc:2069] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.442978 27637 raft_consensus.cc:479] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.444105 27637 leader_election.cc:243] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.444466 27200 tablet_replica.cc:261] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:24.444519 27637 leader_election.cc:243] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.444720 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.444911 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.445096 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.445299 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.445510 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.445725 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.445899 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(7e75ff58c90a4729b884387e271fefa6)
I0619 00:12:24.446071 27200 raft_consensus.cc:2042] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:24.446022 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "02d950f63c9c46d381be1c3b31f5a2e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:24.446254 27200 consensus_peers.cc:456] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:24.446408 26928 ts_tablet_manager.cc:1173] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.446441 27200 consensus_peers.cc:456] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:24.446491 26948 raft_consensus.cc:2784] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.446792 26948 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.216s (vote granted)
I0619 00:12:24.446841 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "02d950f63c9c46d381be1c3b31f5a2e4" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:24.446975 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet c000367cc97d42188f684e5f5df4f9f3 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.447283 26811 raft_consensus.cc:2784] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.447520 26811 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.585s (vote granted)
I0619 00:12:24.448993 27210 tablet_replica.cc:261] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:24.449098 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 7e75ff58c90a4729b884387e271fefa6 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:24.449265 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.449451 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.449658 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.449679 27063 ts_tablet_manager.cc:1173] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.449847 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.450062 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.450256 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.450433 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(c000367cc97d42188f684e5f5df4f9f3)
I0619 00:12:24.450635 27210 raft_consensus.cc:2042] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.450887 27210 raft_consensus.cc:2069] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.451539 27200 raft_consensus.cc:2069] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.452929 26948 raft_consensus.cc:2245] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:24.453985 27013 leader_election.cc:379] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.454262 27013 leader_election.cc:270] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:24.454362 27195 ts_tablet_manager.cc:1028] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
W0619 00:12:24.454473 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.455104 27197 tablet_bootstrap.cc:427] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:24.455427 27197 ts_tablet_manager.cc:1045] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.067s	user 0.005s	sys 0.003s
I0619 00:12:24.455708 27195 tablet_bootstrap.cc:427] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:24.456948 27200 ts_tablet_manager.cc:1369] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.456728 27197 raft_consensus.cc:325] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.457335 27197 raft_consensus.cc:351] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.457522 27197 raft_consensus.cc:628] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:24.458343 27195 tablet_bootstrap.cc:578] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:24.458057 27197 consensus_queue.cc:227] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.458945 27637 raft_consensus.cc:2732] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.051s (election complete)
I0619 00:12:24.459208 27637 raft_consensus.cc:2539] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:24.459719 27210 ts_tablet_manager.cc:1369] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.459503 27637 raft_consensus.cc:593] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:24.460224 27637 consensus_queue.cc:204] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.461760 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.459568 26811 raft_consensus.cc:2245] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:24.462888 27015 leader_election.cc:379] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.464231 27199 ts_tablet_manager.cc:1369] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.470994 27196 consensus_queue.cc:227] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.474308 27196 ts_tablet_manager.cc:1073] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.038s	user 0.002s	sys 0.001s
I0619 00:12:24.474741 27196 ts_tablet_manager.cc:1028] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:24.475102 27196 tablet_bootstrap.cc:427] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:24.476362 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.478678 27196 tablet_bootstrap.cc:578] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:24.486594 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.491330 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.493572 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.497637 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.497782 27195 tablet_bootstrap.cc:427] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:24.498165 27195 ts_tablet_manager.cc:1045] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.043s	user 0.005s	sys 0.000s
I0619 00:12:24.500469 26532 catalog_manager.cc:3786] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
W0619 00:12:24.502396 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.502697 27197 ts_tablet_manager.cc:1073] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.047s	user 0.005s	sys 0.000s
I0619 00:12:24.502655 27195 raft_consensus.cc:325] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.503310 27197 ts_tablet_manager.cc:1028] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:24.503309 27195 raft_consensus.cc:351] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.503485 27195 raft_consensus.cc:628] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:24.503770 27197 tablet_bootstrap.cc:427] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:24.503932 27195 consensus_queue.cc:227] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.505751 27195 ts_tablet_manager.cc:1073] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I0619 00:12:24.507658 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.481967 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.513209 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.517097 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.518164 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.521634 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.522327 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.522619 27637 raft_consensus.cc:457] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.522970 27637 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.604s (starting election)
I0619 00:12:24.523224 27637 raft_consensus.cc:479] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.524322 27637 leader_election.cc:243] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.469458 27200 ts_tablet_manager.cc:1382] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.524736 27200 log.cc:986] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/7e75ff58c90a4729b884387e271fefa6
I0619 00:12:24.525425 27200 ts_tablet_manager.cc:1403] T 7e75ff58c90a4729b884387e271fefa6 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:24.526468 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 7e75ff58c90a4729b884387e271fefa6 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:24.528748 27637 leader_election.cc:243] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.529680 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet bf54a7ce52744b50ac35376a9cda49fa from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:24.529934 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2e2867d8ea324cddb373bcfebb206143" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:24.530380 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2e2867d8ea324cddb373bcfebb206143" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:24.509163 27197 tablet_bootstrap.cc:578] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
W0619 00:12:24.531481 27015 leader_election.cc:306] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Illegal state: must be running to vote when last-logged opid is not known
W0619 00:12:24.531816 27013 leader_election.cc:306] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:24.532063 27013 leader_election.cc:270] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:24.535380 27199 ts_tablet_manager.cc:1382] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.535667 27637 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.654s (election complete)
I0619 00:12:24.538013 27637 raft_consensus.cc:2487] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:24.542929 27199 log.cc:986] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/7e75ff58c90a4729b884387e271fefa6
I0619 00:12:24.544100 27199 ts_tablet_manager.cc:1403] T 7e75ff58c90a4729b884387e271fefa6 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:24.544214 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet bf54a7ce52744b50ac35376a9cda49fa
I0619 00:12:24.544497 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bf54a7ce52744b50ac35376a9cda49fa on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 9015a9008da640a1935c5b52b96cba79)
I0619 00:12:24.544884 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bf54a7ce52744b50ac35376a9cda49fa on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 9015a9008da640a1935c5b52b96cba79)
I0619 00:12:24.545486 26928 tablet_service.cc:800] Processing DeleteTablet for tablet bf54a7ce52744b50ac35376a9cda49fa with delete_type TABLET_DATA_DELETED (replaced by tablet 9015a9008da640a1935c5b52b96cba79) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.545819 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bf54a7ce52744b50ac35376a9cda49fa on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 9015a9008da640a1935c5b52b96cba79)
I0619 00:12:24.545823 26792 tablet_service.cc:800] Processing DeleteTablet for tablet bf54a7ce52744b50ac35376a9cda49fa with delete_type TABLET_DATA_DELETED (replaced by tablet 9015a9008da640a1935c5b52b96cba79) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:24.546735 27063 tablet_service.cc:800] Processing DeleteTablet for tablet bf54a7ce52744b50ac35376a9cda49fa with delete_type TABLET_DATA_DELETED (replaced by tablet 9015a9008da640a1935c5b52b96cba79) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:24.548585 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 7e75ff58c90a4729b884387e271fefa6 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:24.549212 27199 tablet_replica.cc:261] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:24.549487 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.549748 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.549978 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.550199 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.550544 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.551285 27197 tablet_bootstrap.cc:427] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:24.551638 27197 ts_tablet_manager.cc:1045] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.048s	user 0.004s	sys 0.004s
I0619 00:12:24.553056 27197 raft_consensus.cc:325] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.554041 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.554263 27197 raft_consensus.cc:351] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.554744 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.554941 27197 raft_consensus.cc:628] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:24.555454 27199 raft_consensus.cc:2042] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.558077 26928 tablet_service.cc:759] Processing CreateTablet for tablet 9015a9008da640a1935c5b52b96cba79 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.558728 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9015a9008da640a1935c5b52b96cba79. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.558885 27063 tablet_service.cc:759] Processing CreateTablet for tablet 9015a9008da640a1935c5b52b96cba79 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.559494 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9015a9008da640a1935c5b52b96cba79. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.561933 27200 tablet_replica.cc:261] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:24.562206 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.562438 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.562135 26792 tablet_service.cc:759] Processing CreateTablet for tablet 9015a9008da640a1935c5b52b96cba79 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.563155 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.563357 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.563452 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9015a9008da640a1935c5b52b96cba79. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.563567 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.563796 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.563978 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.564146 27200 raft_consensus.cc:2042] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.564396 27200 raft_consensus.cc:2069] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.565798 26532 catalog_manager.cc:4596] ReplaceTablet: tablet bf54a7ce52744b50ac35376a9cda49fa deleted and replaced by tablet 9015a9008da640a1935c5b52b96cba79
I0619 00:12:24.567634 27612 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=80 ignored=1} inserts{seen=36 ignored=0} mutations{seen=71 ignored=0} orphaned_commits=0. Pending: 1 replicates
I0619 00:12:24.568527 27612 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Bootstrap complete.
I0619 00:12:24.570874 27210 ts_tablet_manager.cc:1382] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.571102 27210 log.cc:986] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.571559 27210 ts_tablet_manager.cc:1403] T c000367cc97d42188f684e5f5df4f9f3 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:24.573403 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet c000367cc97d42188f684e5f5df4f9f3 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
W0619 00:12:24.574023 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet c000367cc97d42188f684e5f5df4f9f3 because the tablet was not found. No further retry: Not found: Tablet not found: c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.574770 26948 raft_consensus.cc:1081] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:24.575011 26521 catalog_manager.cc:3071] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): delete failed for tablet c000367cc97d42188f684e5f5df4f9f3 because the tablet was not found. No further retry: Not found: Tablet not found: c000367cc97d42188f684e5f5df4f9f3
I0619 00:12:24.575554 27210 tablet_replica.cc:261] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:24.575850 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.576042 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.576234 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.576139 27637 consensus_queue.cc:963] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:24.576426 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.576675 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.576881 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.577072 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(bf54a7ce52744b50ac35376a9cda49fa)
I0619 00:12:24.577246 27210 raft_consensus.cc:2042] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.577497 27210 raft_consensus.cc:2069] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.576303 26928 ts_tablet_manager.cc:1173] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.582646 27200 ts_tablet_manager.cc:1369] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.583879 26792 ts_tablet_manager.cc:1173] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.584769 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.584261 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet bf54a7ce52744b50ac35376a9cda49fa from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:24.585302 27196 tablet_bootstrap.cc:427] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:24.585636 27196 ts_tablet_manager.cc:1045] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.111s	user 0.007s	sys 0.002s
I0619 00:12:24.587252 27196 raft_consensus.cc:325] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.587924 27196 raft_consensus.cc:351] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.588129 27196 raft_consensus.cc:628] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:24.590864 27063 ts_tablet_manager.cc:1173] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.557832 27199 raft_consensus.cc:2069] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.592324 27195 ts_tablet_manager.cc:1028] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:24.592737 27195 tablet_bootstrap.cc:427] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:24.593020 27196 consensus_queue.cc:227] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.556159 27197 consensus_queue.cc:227] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.594640 27199 ts_tablet_manager.cc:1369] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.595974 27197 ts_tablet_manager.cc:1073] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.044s	user 0.003s	sys 0.000s
I0619 00:12:24.608734 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.595134 27195 tablet_bootstrap.cc:578] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
W0619 00:12:24.612856 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.613062 27200 ts_tablet_manager.cc:1382] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.613301 27200 log.cc:986] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/bf54a7ce52744b50ac35376a9cda49fa
W0619 00:12:24.604213 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.614483 27197 ts_tablet_manager.cc:1028] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:24.610391 27657 raft_consensus.cc:457] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.614742 27657 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.533s (starting election)
I0619 00:12:24.614991 27657 raft_consensus.cc:479] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.615962 27657 leader_election.cc:243] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.616356 27657 leader_election.cc:243] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.617388 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.606160 27210 ts_tablet_manager.cc:1369] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.618515 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2e2867d8ea324cddb373bcfebb206143" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:24.618968 26811 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.617s (vote granted)
I0619 00:12:24.619179 26811 raft_consensus.cc:2245] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:24.619894 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2e2867d8ea324cddb373bcfebb206143" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:24.620363 27083 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.568s (vote granted)
I0619 00:12:24.620573 27083 raft_consensus.cc:2245] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:24.621955 26874 leader_election.cc:379] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.622176 26874 leader_election.cc:270] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
W0619 00:12:24.622287 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.622988 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.623205 26876 leader_election.cc:379] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.624311 27657 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.016s (election complete)
I0619 00:12:24.624545 27657 raft_consensus.cc:2539] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:24.624748 27657 raft_consensus.cc:457] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:24.624924 27657 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.050s (starting election)
I0619 00:12:24.625111 27657 raft_consensus.cc:2784] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
W0619 00:12:24.627462 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.629209 27657 raft_consensus.cc:479] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.630256 27657 leader_election.cc:243] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.630718 27657 leader_election.cc:243] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.631742 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2e2867d8ea324cddb373bcfebb206143" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:24.632138 26811 raft_consensus.cc:2784] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.632365 26811 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.173s (vote granted)
I0619 00:12:24.637073 26812 raft_consensus.cc:1081] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:24.641010 27195 tablet_bootstrap.cc:427] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:24.641386 27195 ts_tablet_manager.cc:1045] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.049s	user 0.004s	sys 0.004s
I0619 00:12:24.643059 26811 raft_consensus.cc:2245] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:24.644596 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.644464 27546 consensus_queue.cc:963] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:24.613806 27200 ts_tablet_manager.cc:1403] T bf54a7ce52744b50ac35376a9cda49fa P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:24.644438 26876 leader_election.cc:379] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.646662 26876 leader_election.cc:270] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:24.647203 27328 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.234s (election complete)
I0619 00:12:24.647429 27328 raft_consensus.cc:2539] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:24.651003 27197 tablet_bootstrap.cc:427] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:24.651854 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.652026 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet bf54a7ce52744b50ac35376a9cda49fa (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.653174 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.654016 27197 tablet_bootstrap.cc:578] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:24.613744 27196 ts_tablet_manager.cc:1073] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.000s
I0619 00:12:24.655642 27196 ts_tablet_manager.cc:1028] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:24.656013 27196 tablet_bootstrap.cc:427] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:24.656987 27328 raft_consensus.cc:593] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:24.633500 27210 ts_tablet_manager.cc:1382] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W0619 00:12:24.658191 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.657529 27328 consensus_queue.cc:204] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.658622 27210 log.cc:986] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/bf54a7ce52744b50ac35376a9cda49fa
I0619 00:12:24.659616 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2e2867d8ea324cddb373bcfebb206143" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:24.660097 27083 raft_consensus.cc:2784] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.660373 27083 raft_consensus.cc:2732] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.416s (vote granted)
I0619 00:12:24.660130 27195 raft_consensus.cc:325] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.660917 27195 raft_consensus.cc:351] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.661128 27195 raft_consensus.cc:628] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:24.661720 27195 consensus_queue.cc:227] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.664541 27083 raft_consensus.cc:2245] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:24.665724 26874 leader_election.cc:379] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.666936 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.673744 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.676195 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.678655 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.679385 27196 tablet_bootstrap.cc:578] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
W0619 00:12:24.683369 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.683938 27210 ts_tablet_manager.cc:1403] T bf54a7ce52744b50ac35376a9cda49fa P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:24.684952 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet bf54a7ce52744b50ac35376a9cda49fa (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.685520 27195 ts_tablet_manager.cc:1073] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.044s	user 0.003s	sys 0.001s
I0619 00:12:24.685981 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.689704 27199 ts_tablet_manager.cc:1382] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.689952 27199 log.cc:986] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/bf54a7ce52744b50ac35376a9cda49fa
I0619 00:12:24.690407 27199 ts_tablet_manager.cc:1403] T bf54a7ce52744b50ac35376a9cda49fa P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:24.691874 27196 tablet_bootstrap.cc:427] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:24.692209 27196 ts_tablet_manager.cc:1045] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.036s	user 0.005s	sys 0.000s
I0619 00:12:24.681699 27612 raft_consensus.cc:325] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:24.693521 27196 raft_consensus.cc:325] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.694146 27196 raft_consensus.cc:351] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.694331 27196 raft_consensus.cc:628] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
W0619 00:12:24.695713 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.695325 27196 consensus_queue.cc:227] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.703788 26532 catalog_manager.cc:3786] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:24.704697 27197 tablet_bootstrap.cc:427] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:24.705049 27197 ts_tablet_manager.cc:1045] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.054s	user 0.005s	sys 0.000s
I0619 00:12:24.706356 27197 raft_consensus.cc:325] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.706980 27197 raft_consensus.cc:351] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.707161 27197 raft_consensus.cc:628] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:24.709602 27197 consensus_queue.cc:227] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.712704 27197 ts_tablet_manager.cc:1073] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I0619 00:12:24.717710 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.719074 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet bf54a7ce52744b50ac35376a9cda49fa (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:24.723117 27196 ts_tablet_manager.cc:1073] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.000s
W0619 00:12:24.723760 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.726341 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.730711 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.731895 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.733976 27612 raft_consensus.cc:628] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555bff44d4f942eaad13f790cca0a696, State: Initialized, Role: FOLLOWER
I0619 00:12:24.734671 27612 consensus_queue.cc:227] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
W0619 00:12:24.740023 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.747598 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.748020 27612 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:24.748718 27612 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:24.749969 27612 sys_catalog.cc:424] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: configured and running, proceeding with master startup.
I0619 00:12:24.770213 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 02d950f63c9c46d381be1c3b31f5a2e4 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:24.780233 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 02d950f63c9c46d381be1c3b31f5a2e4
I0619 00:12:24.780535 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 02d950f63c9c46d381be1c3b31f5a2e4 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:24.780943 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 02d950f63c9c46d381be1c3b31f5a2e4 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:24.781319 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 02d950f63c9c46d381be1c3b31f5a2e4 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:24.781631 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 02d950f63c9c46d381be1c3b31f5a2e4 deleted and replaced by tablet 56d41c14ebcb4acab507b5b87f2aa8db
I0619 00:12:24.783928 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 02d950f63c9c46d381be1c3b31f5a2e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 56d41c14ebcb4acab507b5b87f2aa8db) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:24.783946 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 02d950f63c9c46d381be1c3b31f5a2e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 56d41c14ebcb4acab507b5b87f2aa8db) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:24.784449 27210 tablet_replica.cc:261] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:24.784440 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 02d950f63c9c46d381be1c3b31f5a2e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 56d41c14ebcb4acab507b5b87f2aa8db) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:24.784708 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.784829 27200 tablet_replica.cc:261] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:24.784900 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.784955 27199 tablet_replica.cc:261] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:24.785078 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785071 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785187 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785256 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785261 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785384 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785436 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785466 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785710 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785704 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785704 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785884 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785895 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.785928 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.786054 27210 raft_consensus.cc:2042] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.786110 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.786128 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.786335 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.786319 27210 raft_consensus.cc:2069] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.786365 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.786545 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(02d950f63c9c46d381be1c3b31f5a2e4)
I0619 00:12:24.786510 27200 raft_consensus.cc:2042] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:24.786742 27199 raft_consensus.cc:2042] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:24.786815 27200 raft_consensus.cc:2069] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.786948 27199 consensus_peers.cc:456] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:24.787147 27199 consensus_peers.cc:456] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:24.787495 27199 pending_rounds.cc:63] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:24.787549 27210 ts_tablet_manager.cc:1369] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.787667 27199 pending_rounds.cc:68] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391042914836480 op_type: NO_OP noop_request { }
I0619 00:12:24.787969 27200 ts_tablet_manager.cc:1369] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.787961 27199 raft_consensus.cc:2622] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:24.788178 27199 raft_consensus.cc:2069] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:24.790750 27546 raft_consensus.cc:457] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.790982 27546 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.120s (starting election)
I0619 00:12:24.791227 27546 raft_consensus.cc:479] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.792595 27546 leader_election.cc:243] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.792636 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 02d950f63c9c46d381be1c3b31f5a2e4 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:24.793005 27546 leader_election.cc:243] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.794505 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9015a9008da640a1935c5b52b96cba79" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:24.794963 26812 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.549s (vote granted)
I0619 00:12:24.795181 26812 raft_consensus.cc:2245] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.795675 27199 ts_tablet_manager.cc:1369] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:24.795771 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9015a9008da640a1935c5b52b96cba79" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:24.796237 26948 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.877s (vote granted)
I0619 00:12:24.796445 26948 raft_consensus.cc:2245] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:24.797536 27015 leader_election.cc:379] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.797791 27015 leader_election.cc:270] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:24.798328 27013 leader_election.cc:379] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.800335 26928 tablet_service.cc:759] Processing CreateTablet for tablet 56d41c14ebcb4acab507b5b87f2aa8db (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.800998 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 56d41c14ebcb4acab507b5b87f2aa8db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.801437 27063 tablet_service.cc:759] Processing CreateTablet for tablet 56d41c14ebcb4acab507b5b87f2aa8db (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.802050 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 56d41c14ebcb4acab507b5b87f2aa8db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.802775 27546 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.863s (election complete)
I0619 00:12:24.803015 27546 raft_consensus.cc:2539] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:24.803210 27546 raft_consensus.cc:457] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:24.803395 27546 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.599s (starting election)
I0619 00:12:24.803577 27546 raft_consensus.cc:2784] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.805040 26792 tablet_service.cc:759] Processing CreateTablet for tablet 56d41c14ebcb4acab507b5b87f2aa8db (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:24.805663 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 56d41c14ebcb4acab507b5b87f2aa8db. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:24.807668 27546 raft_consensus.cc:479] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.808620 27546 leader_election.cc:243] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.809118 27546 leader_election.cc:243] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.810256 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9015a9008da640a1935c5b52b96cba79" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:24.810523 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9015a9008da640a1935c5b52b96cba79" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:24.810746 26948 raft_consensus.cc:2784] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.810973 26812 raft_consensus.cc:2784] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.810978 26948 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.026s (vote granted)
I0619 00:12:24.811192 26812 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.636s (vote granted)
I0619 00:12:24.815105 26812 raft_consensus.cc:2245] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:24.815240 26948 raft_consensus.cc:2245] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:24.816076 27015 leader_election.cc:379] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:24.816311 27015 leader_election.cc:270] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:24.816550 27013 leader_election.cc:379] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.816819 27546 raft_consensus.cc:2732] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.583s (election complete)
I0619 00:12:24.817070 27546 raft_consensus.cc:2539] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:24.819733 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.817369 27546 raft_consensus.cc:593] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:24.820912 27546 consensus_queue.cc:204] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.824142 26792 ts_tablet_manager.cc:1173] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.826159 27196 ts_tablet_manager.cc:1028] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:24.826503 27196 tablet_bootstrap.cc:427] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:24.825605 27063 ts_tablet_manager.cc:1173] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.827069 27195 ts_tablet_manager.cc:1028] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:24.827384 27195 tablet_bootstrap.cc:427] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:24.828864 27196 tablet_bootstrap.cc:578] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:24.831640 26928 ts_tablet_manager.cc:1173] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:24.832077 27197 ts_tablet_manager.cc:1028] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:24.832427 27197 tablet_bootstrap.cc:427] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:24.832790 27210 ts_tablet_manager.cc:1382] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.833030 27210 log.cc:986] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/02d950f63c9c46d381be1c3b31f5a2e4
I0619 00:12:24.833515 27210 ts_tablet_manager.cc:1403] T 02d950f63c9c46d381be1c3b31f5a2e4 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:24.835003 26532 catalog_manager.cc:3786] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
W0619 00:12:24.837561 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.841359 27195 tablet_bootstrap.cc:578] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:24.845386 27197 tablet_bootstrap.cc:578] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:24.847323 27200 ts_tablet_manager.cc:1382] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:24.848505 27200 log.cc:986] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/02d950f63c9c46d381be1c3b31f5a2e4
I0619 00:12:24.848709 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 02d950f63c9c46d381be1c3b31f5a2e4 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:24.849093 27200 ts_tablet_manager.cc:1403] T 02d950f63c9c46d381be1c3b31f5a2e4 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:24.852792 27199 ts_tablet_manager.cc:1382] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:24.853049 27199 log.cc:986] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/02d950f63c9c46d381be1c3b31f5a2e4
I0619 00:12:24.854307 27196 tablet_bootstrap.cc:427] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:24.854634 27196 ts_tablet_manager.cc:1045] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.003s
I0619 00:12:24.855960 27196 raft_consensus.cc:325] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.856525 27196 raft_consensus.cc:351] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.856726 27196 raft_consensus.cc:628] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:24.857311 27196 consensus_queue.cc:227] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.858505 27197 tablet_bootstrap.cc:427] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:24.858840 27197 ts_tablet_manager.cc:1045] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.027s	user 0.005s	sys 0.000s
I0619 00:12:24.860218 27197 raft_consensus.cc:325] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.860839 27197 raft_consensus.cc:351] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.861016 27197 raft_consensus.cc:628] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:24.861937 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 02d950f63c9c46d381be1c3b31f5a2e4 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:24.864539 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.864720 27196 ts_tablet_manager.cc:1073] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I0619 00:12:24.861441 27197 consensus_queue.cc:227] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:24.869555 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.871013 27195 tablet_bootstrap.cc:427] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:24.871392 27195 ts_tablet_manager.cc:1045] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.044s	user 0.006s	sys 0.002s
I0619 00:12:24.871811 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.872813 27195 raft_consensus.cc:325] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.873497 27195 raft_consensus.cc:351] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:24.873734 27195 raft_consensus.cc:628] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:24.874294 27195 consensus_queue.cc:227] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.875228 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.877774 27197 ts_tablet_manager.cc:1073] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.000s
I0619 00:12:24.877794 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.879803 27195 ts_tablet_manager.cc:1073] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
W0619 00:12:24.880523 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:24.879369 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.884313 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.885082 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.887418 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:24.891541 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.853616 27199 ts_tablet_manager.cc:1403] T 02d950f63c9c46d381be1c3b31f5a2e4 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:24.893673 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 02d950f63c9c46d381be1c3b31f5a2e4 (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
W0619 00:12:24.893820 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.915503 27424 raft_consensus.cc:457] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:24.915818 27424 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.981s (starting election)
I0619 00:12:24.916064 27424 raft_consensus.cc:479] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.917088 27424 leader_election.cc:243] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.917476 27424 leader_election.cc:243] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.919293 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "56d41c14ebcb4acab507b5b87f2aa8db" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:24.919795 27083 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.136s (vote granted)
I0619 00:12:24.920008 27083 raft_consensus.cc:2245] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:24.920639 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "56d41c14ebcb4acab507b5b87f2aa8db" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:24.921130 26948 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.139s (vote granted)
I0619 00:12:24.921342 26948 raft_consensus.cc:2245] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:24.922797 26732 leader_election.cc:379] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.923030 26732 leader_election.cc:270] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:24.923593 26732 leader_election.cc:379] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.924046 27424 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.119s (election complete)
I0619 00:12:24.924291 27424 raft_consensus.cc:2539] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:24.924489 27424 raft_consensus.cc:457] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:24.924705 27424 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.651s (starting election)
I0619 00:12:24.924901 27424 raft_consensus.cc:2784] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.929229 27424 raft_consensus.cc:479] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.931442 27424 leader_election.cc:243] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.937901 27424 leader_election.cc:243] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.938002 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "56d41c14ebcb4acab507b5b87f2aa8db" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:24.938418 26948 raft_consensus.cc:2784] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.938676 26948 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.978s (vote granted)
I0619 00:12:24.942523 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "56d41c14ebcb4acab507b5b87f2aa8db" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:24.942987 27083 raft_consensus.cc:2784] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:24.943228 27083 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.236s (vote granted)
I0619 00:12:24.947515 27083 raft_consensus.cc:2245] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:24.949290 26948 raft_consensus.cc:2245] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:24.950299 26732 leader_election.cc:379] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:24.950538 26732 leader_election.cc:270] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:24.951441 27424 raft_consensus.cc:2732] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.102s (election complete)
I0619 00:12:24.952136 27424 raft_consensus.cc:2539] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:24.952849 26732 leader_election.cc:379] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:24.959494 27424 raft_consensus.cc:593] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:24.959954 27424 consensus_queue.cc:204] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:24.960479 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:24.968029 26533 catalog_manager.cc:3786] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
W0619 00:12:24.968787 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:24.992549 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 9015a9008da640a1935c5b52b96cba79 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:25.002800 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet 2e2867d8ea324cddb373bcfebb206143 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:25.005534 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 9015a9008da640a1935c5b52b96cba79
I0619 00:12:25.005843 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9015a9008da640a1935c5b52b96cba79 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.006224 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9015a9008da640a1935c5b52b96cba79 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.006589 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 9015a9008da640a1935c5b52b96cba79 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.006916 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 9015a9008da640a1935c5b52b96cba79 deleted and replaced by tablet bef7d1a41629429bb18d6f6e859592c5
I0619 00:12:25.006845 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 9015a9008da640a1935c5b52b96cba79 with delete_type TABLET_DATA_DELETED (replaced by tablet bef7d1a41629429bb18d6f6e859592c5) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:25.007320 27210 tablet_replica.cc:261] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:25.007541 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.007745 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.007913 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.008081 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.008278 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.008170 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 9015a9008da640a1935c5b52b96cba79 with delete_type TABLET_DATA_DELETED (replaced by tablet bef7d1a41629429bb18d6f6e859592c5) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:25.008451 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.008654 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.008699 27200 tablet_replica.cc:261] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:25.008821 27210 raft_consensus.cc:2042] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.008963 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.009068 27210 raft_consensus.cc:2069] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.009160 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.009366 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.009562 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.009810 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.010007 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.010191 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.010380 27200 raft_consensus.cc:2042] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.010484 27210 ts_tablet_manager.cc:1369] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.010689 27200 raft_consensus.cc:2069] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.012080 27200 ts_tablet_manager.cc:1369] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.017359 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 2e2867d8ea324cddb373bcfebb206143
I0619 00:12:25.017660 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2e2867d8ea324cddb373bcfebb206143 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 1764162224124e32922728fe10894601)
I0619 00:12:25.018026 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2e2867d8ea324cddb373bcfebb206143 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 1764162224124e32922728fe10894601)
I0619 00:12:25.018385 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 2e2867d8ea324cddb373bcfebb206143 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 1764162224124e32922728fe10894601)
I0619 00:12:25.018708 26532 catalog_manager.cc:4596] ReplaceTablet: tablet 2e2867d8ea324cddb373bcfebb206143 deleted and replaced by tablet 1764162224124e32922728fe10894601
I0619 00:12:25.019503 27210 ts_tablet_manager.cc:1382] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.019735 27210 log.cc:986] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/9015a9008da640a1935c5b52b96cba79
I0619 00:12:25.019861 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 2e2867d8ea324cddb373bcfebb206143 with delete_type TABLET_DATA_DELETED (replaced by tablet 1764162224124e32922728fe10894601) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:25.020222 27210 ts_tablet_manager.cc:1403] T 9015a9008da640a1935c5b52b96cba79 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:25.021164 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 2e2867d8ea324cddb373bcfebb206143 with delete_type TABLET_DATA_DELETED (replaced by tablet 1764162224124e32922728fe10894601) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:25.025482 27200 ts_tablet_manager.cc:1382] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.029464 27210 tablet_replica.cc:261] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:25.029644 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 9015a9008da640a1935c5b52b96cba79 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:25.031373 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.031600 27200 log.cc:986] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/9015a9008da640a1935c5b52b96cba79
I0619 00:12:25.031822 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.032016 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.032202 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.032220 27200 ts_tablet_manager.cc:1403] T 9015a9008da640a1935c5b52b96cba79 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:25.032426 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.032630 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.032846 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.033020 27210 raft_consensus.cc:2042] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:25.033221 27210 consensus_peers.cc:456] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:25.033421 27210 consensus_peers.cc:456] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:25.034745 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 9015a9008da640a1935c5b52b96cba79 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:25.036995 26792 tablet_service.cc:759] Processing CreateTablet for tablet bef7d1a41629429bb18d6f6e859592c5 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.037546 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bef7d1a41629429bb18d6f6e859592c5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.038106 27210 pending_rounds.cc:63] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:25.038249 27210 pending_rounds.cc:68] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391043724587008 op_type: NO_OP noop_request { }
I0619 00:12:25.038731 26928 tablet_service.cc:759] Processing CreateTablet for tablet bef7d1a41629429bb18d6f6e859592c5 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.039157 27210 raft_consensus.cc:2622] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:25.039382 27210 raft_consensus.cc:2069] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.039496 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bef7d1a41629429bb18d6f6e859592c5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.042260 27210 ts_tablet_manager.cc:1369] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.050323 26791 tablet_service.cc:759] Processing CreateTablet for tablet 1764162224124e32922728fe10894601 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.050993 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1764162224124e32922728fe10894601. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.048948 26929 tablet_service.cc:759] Processing CreateTablet for tablet 1764162224124e32922728fe10894601 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.051934 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1764162224124e32922728fe10894601. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.058923 27200 tablet_replica.cc:261] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:25.059244 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.059463 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.059696 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.059912 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.060163 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.060386 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.060592 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.060822 27200 raft_consensus.cc:2042] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.061117 27200 raft_consensus.cc:2069] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.062466 26929 ts_tablet_manager.cc:1173] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.062932 27197 ts_tablet_manager.cc:1028] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:25.063325 27197 tablet_bootstrap.cc:427] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:25.065840 27197 tablet_bootstrap.cc:578] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:25.065647 27062 tablet_service.cc:800] Processing DeleteTablet for tablet 2e2867d8ea324cddb373bcfebb206143 with delete_type TABLET_DATA_DELETED (replaced by tablet 1764162224124e32922728fe10894601) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:25.071107 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 9015a9008da640a1935c5b52b96cba79 with delete_type TABLET_DATA_DELETED (replaced by tablet bef7d1a41629429bb18d6f6e859592c5) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:25.072299 26792 ts_tablet_manager.cc:1173] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.072902 27199 tablet_replica.cc:261] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:25.073249 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.073436 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.073635 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.073639 27061 tablet_service.cc:759] Processing CreateTablet for tablet bef7d1a41629429bb18d6f6e859592c5 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.073810 27197 tablet_bootstrap.cc:427] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:25.073863 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.074086 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.074141 27197 ts_tablet_manager.cc:1045] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.001s
I0619 00:12:25.074296 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.074470 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(2e2867d8ea324cddb373bcfebb206143)
I0619 00:12:25.074698 27199 raft_consensus.cc:2042] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.074818 27061 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bef7d1a41629429bb18d6f6e859592c5. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.074995 27199 raft_consensus.cc:2069] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.075464 27197 raft_consensus.cc:325] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.076079 27197 raft_consensus.cc:351] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.076256 27197 raft_consensus.cc:628] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:25.076748 27199 ts_tablet_manager.cc:1369] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.077061 27197 consensus_queue.cc:227] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.078160 27063 tablet_service.cc:759] Processing CreateTablet for tablet 1764162224124e32922728fe10894601 (table=default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.078800 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1764162224124e32922728fe10894601. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.080014 27197 ts_tablet_manager.cc:1073] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
I0619 00:12:25.083503 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.084321 26791 ts_tablet_manager.cc:1173] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.085438 27196 ts_tablet_manager.cc:1028] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:25.085798 27196 tablet_bootstrap.cc:427] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:25.088793 27196 tablet_bootstrap.cc:578] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
W0619 00:12:25.091208 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.093531 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.094431 27063 ts_tablet_manager.cc:1173] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.094894 27195 ts_tablet_manager.cc:1028] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:25.095312 27195 tablet_bootstrap.cc:427] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:25.097637 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.098182 26928 ts_tablet_manager.cc:1173] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.098942 27197 ts_tablet_manager.cc:1028] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:25.099361 27197 tablet_bootstrap.cc:427] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:25.101636 27197 tablet_bootstrap.cc:578] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:25.102530 27200 ts_tablet_manager.cc:1369] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.106377 27199 ts_tablet_manager.cc:1382] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.106655 27199 log.cc:986] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/2e2867d8ea324cddb373bcfebb206143
I0619 00:12:25.108039 27195 tablet_bootstrap.cc:578] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:25.109721 27196 tablet_bootstrap.cc:427] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:25.110069 27196 ts_tablet_manager.cc:1045] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.024s	user 0.002s	sys 0.010s
I0619 00:12:25.111541 27196 raft_consensus.cc:325] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.112176 27196 raft_consensus.cc:351] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.112391 27196 raft_consensus.cc:628] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:25.112967 27196 consensus_queue.cc:227] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.116272 27199 ts_tablet_manager.cc:1403] T 2e2867d8ea324cddb373bcfebb206143 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:25.117097 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.119829 27196 ts_tablet_manager.cc:1073] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I0619 00:12:25.120326 27196 ts_tablet_manager.cc:1028] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:25.120414 27061 ts_tablet_manager.cc:1173] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:25.123245 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.123407 27200 ts_tablet_manager.cc:1382] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.123663 27200 log.cc:986] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/2e2867d8ea324cddb373bcfebb206143
I0619 00:12:25.124333 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 2e2867d8ea324cddb373bcfebb206143 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:25.125380 27199 tablet_replica.cc:261] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:25.128252 27210 ts_tablet_manager.cc:1382] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:25.128707 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.129278 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.129539 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.129801 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.130231 27195 tablet_bootstrap.cc:427] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:25.130580 27195 ts_tablet_manager.cc:1045] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.035s	user 0.005s	sys 0.006s
I0619 00:12:25.131745 27200 ts_tablet_manager.cc:1403] T 2e2867d8ea324cddb373bcfebb206143 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:25.125528 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.132252 27210 log.cc:986] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/2e2867d8ea324cddb373bcfebb206143
I0619 00:12:25.134253 27210 ts_tablet_manager.cc:1403] T 2e2867d8ea324cddb373bcfebb206143 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:25.134428 27195 raft_consensus.cc:325] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.135025 27195 raft_consensus.cc:351] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.135414 27195 raft_consensus.cc:628] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:25.132794 27197 tablet_bootstrap.cc:427] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:25.135962 27197 ts_tablet_manager.cc:1045] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.037s	user 0.004s	sys 0.005s
W0619 00:12:25.136138 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.130489 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.136667 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.136934 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(9015a9008da640a1935c5b52b96cba79)
I0619 00:12:25.137145 27199 raft_consensus.cc:2042] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:25.136607 27195 consensus_queue.cc:227] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.137468 27199 consensus_peers.cc:456] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:25.137351 27197 raft_consensus.cc:325] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.124372 27196 tablet_bootstrap.cc:427] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:25.138000 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 2e2867d8ea324cddb373bcfebb206143 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:25.138135 27197 raft_consensus.cc:351] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.138384 27197 raft_consensus.cc:628] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:25.138968 27197 consensus_queue.cc:227] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.142340 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.139289 27195 ts_tablet_manager.cc:1073] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.002s
I0619 00:12:25.146570 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.146924 27195 ts_tablet_manager.cc:1028] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:25.147264 27195 tablet_bootstrap.cc:427] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
W0619 00:12:25.147795 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.148164 27199 consensus_peers.cc:456] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:25.148540 27199 pending_rounds.cc:63] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Trying to abort 1 pending transactions.
I0619 00:12:25.148748 27199 pending_rounds.cc:68] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391044399325184 op_type: NO_OP noop_request { }
I0619 00:12:25.149698 27196 tablet_bootstrap.cc:578] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:25.149979 27199 raft_consensus.cc:2622] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:25.150218 27199 raft_consensus.cc:2069] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
W0619 00:12:25.150775 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.152977 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.153240 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 2e2867d8ea324cddb373bcfebb206143 (table default.table_4a6e4e28e18547719a48db319fc95750 [id=4accbe5f802d41238340171a9b94219b]) successfully deleted
I0619 00:12:25.154628 27199 ts_tablet_manager.cc:1369] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.157593 27196 tablet_bootstrap.cc:427] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:25.157925 27196 ts_tablet_manager.cc:1045] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.037s	user 0.003s	sys 0.004s
I0619 00:12:24.748316 27673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:25.159977 27673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:25.160094 27583 raft_consensus.cc:1026] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.141]   Dedup: 1.81->[1.82-1.141]
I0619 00:12:25.165029 27195 tablet_bootstrap.cc:578] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:25.165045 27197 ts_tablet_manager.cc:1073] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.029s	user 0.003s	sys 0.001s
W0619 00:12:25.165045 27000 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.167256 27196 raft_consensus.cc:325] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.167935 27196 raft_consensus.cc:351] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.168144 27196 raft_consensus.cc:628] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:25.168728 27196 consensus_queue.cc:227] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.170162 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:25.177320 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.180362 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:25.185190 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
W0619 00:12:25.186715 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.187178 27196 ts_tablet_manager.cc:1073] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.029s	user 0.002s	sys 0.001s
I0619 00:12:25.189066 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
W0619 00:12:25.193294 26857 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.195237 27199 ts_tablet_manager.cc:1382] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:25.195524 27199 log.cc:986] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/9015a9008da640a1935c5b52b96cba79
I0619 00:12:25.196148 27199 ts_tablet_manager.cc:1403] T 9015a9008da640a1935c5b52b96cba79 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:25.199482 27195 tablet_bootstrap.cc:427] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:25.199838 27195 ts_tablet_manager.cc:1045] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.053s	user 0.004s	sys 0.001s
I0619 00:12:25.201241 27195 raft_consensus.cc:325] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.201870 27195 raft_consensus.cc:351] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.202047 27195 raft_consensus.cc:628] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:25.202579 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 9015a9008da640a1935c5b52b96cba79 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:25.203197 27195 consensus_queue.cc:227] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.208510 27195 ts_tablet_manager.cc:1073] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I0619 00:12:25.211482 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.214401 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet e9045aa5906b41e2ae3e437b52f0b417 from {username='jenkins'} at 127.24.119.62:33747
W0619 00:12:25.218981 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.221326 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.226536 26532 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet e9045aa5906b41e2ae3e437b52f0b417
I0619 00:12:25.226845 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9045aa5906b41e2ae3e437b52f0b417 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:25.227221 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9045aa5906b41e2ae3e437b52f0b417 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:25.228255 26928 tablet_service.cc:800] Processing DeleteTablet for tablet e9045aa5906b41e2ae3e437b52f0b417 with delete_type TABLET_DATA_DELETED (replaced by tablet e63eb8276d6d4a03b30d54126e965dbd) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:25.228780 27210 tablet_replica.cc:261] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:25.229015 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.229205 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.229378 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.229575 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.229805 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.230026 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.230199 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.230361 27210 raft_consensus.cc:2042] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.230648 27210 raft_consensus.cc:2069] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.231948 26792 tablet_service.cc:800] Processing DeleteTablet for tablet e9045aa5906b41e2ae3e437b52f0b417 with delete_type TABLET_DATA_DELETED (replaced by tablet e63eb8276d6d4a03b30d54126e965dbd) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:25.232430 27200 tablet_replica.cc:261] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:25.232679 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.232867 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.233038 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.233026 27210 ts_tablet_manager.cc:1369] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.233211 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.233417 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.233629 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.233808 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.233976 27200 raft_consensus.cc:2042] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.234243 27200 raft_consensus.cc:2069] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.235306 26532 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e9045aa5906b41e2ae3e437b52f0b417 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:25.237244 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet 56d41c14ebcb4acab507b5b87f2aa8db from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:25.237921 26532 catalog_manager.cc:4596] ReplaceTablet: tablet e9045aa5906b41e2ae3e437b52f0b417 deleted and replaced by tablet e63eb8276d6d4a03b30d54126e965dbd
I0619 00:12:25.243800 27061 tablet_service.cc:800] Processing DeleteTablet for tablet e9045aa5906b41e2ae3e437b52f0b417 with delete_type TABLET_DATA_DELETED (replaced by tablet e63eb8276d6d4a03b30d54126e965dbd) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:25.244280 27199 tablet_replica.cc:261] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:25.244519 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.244738 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.244923 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.245103 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.245316 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.245513 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.245723 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(e9045aa5906b41e2ae3e437b52f0b417)
I0619 00:12:25.245898 27199 raft_consensus.cc:2042] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:25.246090 27199 consensus_peers.cc:456] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:25.246289 27199 consensus_peers.cc:456] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:25.247208 27199 raft_consensus.cc:2069] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.249397 27210 ts_tablet_manager.cc:1382] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:25.249642 27210 log.cc:986] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/e9045aa5906b41e2ae3e437b52f0b417
I0619 00:12:25.250272 27210 ts_tablet_manager.cc:1403] T e9045aa5906b41e2ae3e437b52f0b417 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:25.252140 27328 raft_consensus.cc:457] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:25.252352 27328 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.820s (starting election)
I0619 00:12:25.252382 26792 tablet_service.cc:759] Processing CreateTablet for tablet e63eb8276d6d4a03b30d54126e965dbd (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.252581 27328 raft_consensus.cc:479] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.253032 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e63eb8276d6d4a03b30d54126e965dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.253162 26928 tablet_service.cc:759] Processing CreateTablet for tablet e63eb8276d6d4a03b30d54126e965dbd (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.253530 27328 leader_election.cc:243] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.253777 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e63eb8276d6d4a03b30d54126e965dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.253947 27328 leader_election.cc:243] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.256825 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bef7d1a41629429bb18d6f6e859592c5" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:25.257292 26812 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.938s (vote granted)
I0619 00:12:25.257378 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet e9045aa5906b41e2ae3e437b52f0b417 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:25.257514 26812 raft_consensus.cc:2245] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:25.258705 26876 leader_election.cc:379] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.258934 26876 leader_election.cc:270] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:25.259399 27328 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.664s (election complete)
I0619 00:12:25.259610 27328 raft_consensus.cc:2539] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:25.259816 27328 raft_consensus.cc:457] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:25.260013 27328 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.748s (starting election)
I0619 00:12:25.260190 27328 raft_consensus.cc:2784] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.261445 27200 ts_tablet_manager.cc:1369] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.262387 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 56d41c14ebcb4acab507b5b87f2aa8db
I0619 00:12:25.262704 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 56d41c14ebcb4acab507b5b87f2aa8db on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 912bbbfd578c495ebd11e1eb0d170e7e)
I0619 00:12:25.263095 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 56d41c14ebcb4acab507b5b87f2aa8db on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 912bbbfd578c495ebd11e1eb0d170e7e)
I0619 00:12:25.263456 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 56d41c14ebcb4acab507b5b87f2aa8db on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 912bbbfd578c495ebd11e1eb0d170e7e)
I0619 00:12:25.263620 26929 tablet_service.cc:800] Processing DeleteTablet for tablet 56d41c14ebcb4acab507b5b87f2aa8db with delete_type TABLET_DATA_DELETED (replaced by tablet 912bbbfd578c495ebd11e1eb0d170e7e) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:25.263792 26533 catalog_manager.cc:4596] ReplaceTablet: tablet 56d41c14ebcb4acab507b5b87f2aa8db deleted and replaced by tablet 912bbbfd578c495ebd11e1eb0d170e7e
I0619 00:12:25.265064 26791 tablet_service.cc:800] Processing DeleteTablet for tablet 56d41c14ebcb4acab507b5b87f2aa8db with delete_type TABLET_DATA_DELETED (replaced by tablet 912bbbfd578c495ebd11e1eb0d170e7e) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:25.266436 27328 raft_consensus.cc:479] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.268530 27328 leader_election.cc:243] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.270084 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bef7d1a41629429bb18d6f6e859592c5" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:25.270498 26812 raft_consensus.cc:2784] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.270757 26812 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.657s (vote granted)
I0619 00:12:25.271234 27328 leader_election.cc:243] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.272334 27210 tablet_replica.cc:261] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:25.272632 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.272821 27689 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.272887 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.273126 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.277371 26812 raft_consensus.cc:2245] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:25.280004 26791 tablet_service.cc:759] Processing CreateTablet for tablet 912bbbfd578c495ebd11e1eb0d170e7e (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.280625 26791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 912bbbfd578c495ebd11e1eb0d170e7e. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:25.281600 27133 heartbeater.cc:587] Failed to heartbeat to 127.24.119.60:33918: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.281610 27691 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I0619 00:12:25.282846 27061 tablet_service.cc:759] Processing CreateTablet for tablet e63eb8276d6d4a03b30d54126e965dbd (table=default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.283505 27061 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e63eb8276d6d4a03b30d54126e965dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.283618 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bef7d1a41629429bb18d6f6e859592c5" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:25.284111 27083 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.226s (vote granted)
I0619 00:12:25.284369 27083 raft_consensus.cc:2245] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:25.285131 27063 tablet_service.cc:800] Processing DeleteTablet for tablet 56d41c14ebcb4acab507b5b87f2aa8db with delete_type TABLET_DATA_DELETED (replaced by tablet 912bbbfd578c495ebd11e1eb0d170e7e) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:25.286183 27063 tablet_service.cc:759] Processing CreateTablet for tablet 912bbbfd578c495ebd11e1eb0d170e7e (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.286742 27063 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 912bbbfd578c495ebd11e1eb0d170e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.289314 26791 ts_tablet_manager.cc:1173] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.289331 27199 ts_tablet_manager.cc:1369] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.290768 27474 master_main.cc:86] Master server successfully started.
I0619 00:12:25.291191 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bef7d1a41629429bb18d6f6e859592c5" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:25.291631 27083 raft_consensus.cc:2784] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.291858 27083 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.689s (vote granted)
I0619 00:12:25.291972 27196 ts_tablet_manager.cc:1028] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:25.292306 27196 tablet_bootstrap.cc:427] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:25.294539 27196 tablet_bootstrap.cc:578] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:25.296334 27083 raft_consensus.cc:2245] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:25.300838 26792 ts_tablet_manager.cc:1173] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.301414 27061 ts_tablet_manager.cc:1173] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.305549 27196 tablet_bootstrap.cc:427] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:25.278404 26876 leader_election.cc:379] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.305835 26876 leader_election.cc:270] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:25.305930 27196 ts_tablet_manager.cc:1045] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.004s
I0619 00:12:25.279243 26929 tablet_service.cc:759] Processing CreateTablet for tablet 912bbbfd578c495ebd11e1eb0d170e7e (table=default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.306720 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 912bbbfd578c495ebd11e1eb0d170e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.307361 27196 raft_consensus.cc:325] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.308023 27196 raft_consensus.cc:351] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.308230 27196 raft_consensus.cc:628] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:25.291606 26874 leader_election.cc:379] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.308821 27196 consensus_queue.cc:227] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.310003 26874 leader_election.cc:379] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.313777 27328 raft_consensus.cc:2732] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.918s (election complete)
I0619 00:12:25.314031 27328 raft_consensus.cc:2539] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:25.314484 26928 ts_tablet_manager.cc:1173] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.314934 27197 ts_tablet_manager.cc:1028] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:25.315268 27197 tablet_bootstrap.cc:427] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:25.316332 26857 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.317153 26929 ts_tablet_manager.cc:1173] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.317503 27197 tablet_bootstrap.cc:578] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:25.317905 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.318131 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.319389 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.319579 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.319782 27210 raft_consensus.cc:2042] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.320998 27063 ts_tablet_manager.cc:1173] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.321761 27195 ts_tablet_manager.cc:1028] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:25.322111 27195 tablet_bootstrap.cc:427] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:25.324374 27195 tablet_bootstrap.cc:578] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:25.324936 27196 ts_tablet_manager.cc:1073] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.000s
I0619 00:12:25.325266 27196 ts_tablet_manager.cc:1028] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:25.325677 27196 tablet_bootstrap.cc:427] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:25.327785 27210 raft_consensus.cc:2069] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.328255 27196 tablet_bootstrap.cc:578] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:25.329103 27424 raft_consensus.cc:457] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:25.329339 27424 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.750s (starting election)
I0619 00:12:25.329560 27424 raft_consensus.cc:479] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.329982 27328 raft_consensus.cc:593] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:25.330518 27424 leader_election.cc:243] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.330745 27200 ts_tablet_manager.cc:1382] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:25.330974 27200 log.cc:986] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/e9045aa5906b41e2ae3e437b52f0b417
I0619 00:12:25.330426 27328 consensus_queue.cc:204] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.331331 27637 raft_consensus.cc:457] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:25.331620 27637 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.691s (starting election)
I0619 00:12:25.331858 27637 raft_consensus.cc:479] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.333317 27637 leader_election.cc:243] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.333729 27637 leader_election.cc:243] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.333976 27210 ts_tablet_manager.cc:1369] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.334292 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "912bbbfd578c495ebd11e1eb0d170e7e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:25.336050 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1764162224124e32922728fe10894601" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:25.336470 26948 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.935s (vote granted)
I0619 00:12:25.336707 26948 raft_consensus.cc:2245] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:25.337718 27013 leader_election.cc:379] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.340970 27000 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.342469 27197 tablet_bootstrap.cc:427] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:25.342819 27197 ts_tablet_manager.cc:1045] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.028s	user 0.006s	sys 0.002s
I0619 00:12:25.344135 27197 raft_consensus.cc:325] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.344786 27197 raft_consensus.cc:351] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.344991 27197 raft_consensus.cc:628] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:25.345432 27197 consensus_queue.cc:227] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.327641 27199 ts_tablet_manager.cc:1382] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:25.346200 27199 log.cc:986] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/e9045aa5906b41e2ae3e437b52f0b417
I0619 00:12:25.330929 27424 leader_election.cc:243] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.347018 26533 catalog_manager.cc:3786] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:25.335175 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1764162224124e32922728fe10894601" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:25.348129 26812 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.030s (vote granted)
I0619 00:12:25.348356 26812 raft_consensus.cc:2245] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
W0619 00:12:25.335952 26732 leader_election.cc:306] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:25.346818 27199 ts_tablet_manager.cc:1403] T e9045aa5906b41e2ae3e437b52f0b417 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:25.352617 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "912bbbfd578c495ebd11e1eb0d170e7e" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:25.358497 27200 ts_tablet_manager.cc:1403] T e9045aa5906b41e2ae3e437b52f0b417 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:25.358642 27197 ts_tablet_manager.cc:1073] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
I0619 00:12:25.359067 27197 ts_tablet_manager.cc:1028] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:25.359406 27197 tablet_bootstrap.cc:427] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:25.359835 27199 tablet_replica.cc:261] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:25.360110 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.360334 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.361619 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.361995 27197 tablet_bootstrap.cc:578] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:25.362668 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(56d41c14ebcb4acab507b5b87f2aa8db)
W0619 00:12:25.362978 26732 consensus_peers.cc:434] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Couldn't send request to peer 1f719e8be9c9447a9acd40728c0a5710 for tablet 56d41c14ebcb4acab507b5b87f2aa8db. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:25.363389 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.363622 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.337929 27013 leader_election.cc:270] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:25.364125 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.364322 27199 raft_consensus.cc:2042] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.364481 27637 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.917s (election complete)
I0619 00:12:25.364719 27015 leader_election.cc:379] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.365134 27196 tablet_bootstrap.cc:427] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:25.364781 27637 raft_consensus.cc:2539] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:25.365489 27196 ts_tablet_manager.cc:1045] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.040s	user 0.005s	sys 0.001s
I0619 00:12:25.365532 27637 raft_consensus.cc:457] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:25.365753 27637 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.947s (starting election)
I0619 00:12:25.365960 27637 raft_consensus.cc:2784] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.367285 27196 raft_consensus.cc:325] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.368070 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet e9045aa5906b41e2ae3e437b52f0b417 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:25.367911 27196 raft_consensus.cc:351] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.368585 27196 raft_consensus.cc:628] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:25.369138 27200 tablet_replica.cc:261] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:25.371462 27210 ts_tablet_manager.cc:1382] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.371765 27210 log.cc:986] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/56d41c14ebcb4acab507b5b87f2aa8db
I0619 00:12:25.371963 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet e9045aa5906b41e2ae3e437b52f0b417 (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:25.372241 27210 ts_tablet_manager.cc:1403] T 56d41c14ebcb4acab507b5b87f2aa8db P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:25.373446 27195 tablet_bootstrap.cc:427] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:25.364580 27199 raft_consensus.cc:2069] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.373837 27195 ts_tablet_manager.cc:1045] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.052s	user 0.004s	sys 0.003s
I0619 00:12:25.374900 27199 ts_tablet_manager.cc:1369] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.375147 27195 raft_consensus.cc:325] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.375115 27637 raft_consensus.cc:479] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.375820 27195 raft_consensus.cc:351] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.376052 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.376015 27195 raft_consensus.cc:628] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:25.376436 27195 consensus_queue.cc:227] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
W0619 00:12:25.371646 26732 leader_election.cc:306] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:25.377185 26732 leader_election.cc:270] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost.
I0619 00:12:25.378015 27424 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.127s (election complete)
I0619 00:12:25.378232 27424 raft_consensus.cc:2487] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I0619 00:12:25.378515 27195 ts_tablet_manager.cc:1073] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I0619 00:12:25.378880 27195 ts_tablet_manager.cc:1028] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:25.379204 27195 tablet_bootstrap.cc:427] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:25.380023 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.380254 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.380820 27637 leader_election.cc:243] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.384683 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1764162224124e32922728fe10894601" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:25.385114 26948 raft_consensus.cc:2784] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.385349 26948 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.969s (vote granted)
I0619 00:12:25.386281 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 56d41c14ebcb4acab507b5b87f2aa8db (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:25.387822 27637 leader_election.cc:243] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.382989 27133 heartbeater.cc:346] Connected to a master server at 127.24.119.60:33918
I0619 00:12:25.393626 27197 tablet_bootstrap.cc:427] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:25.394013 27197 ts_tablet_manager.cc:1045] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.035s	user 0.004s	sys 0.001s
I0619 00:12:25.394201 27199 ts_tablet_manager.cc:1382] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.394799 26948 raft_consensus.cc:2245] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:25.395144 27199 log.cc:986] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/56d41c14ebcb4acab507b5b87f2aa8db
I0619 00:12:25.395915 27013 leader_election.cc:379] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.381346 27195 tablet_bootstrap.cc:578] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:25.396214 27013 leader_election.cc:270] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:25.396404 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.392518 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1764162224124e32922728fe10894601" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:25.396661 26812 raft_consensus.cc:2784] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.396893 26812 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.016s (vote granted)
I0619 00:12:25.397142 27199 ts_tablet_manager.cc:1403] T 56d41c14ebcb4acab507b5b87f2aa8db P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:25.399029 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 56d41c14ebcb4acab507b5b87f2aa8db (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:25.369029 27196 consensus_queue.cc:227] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.402777 26812 raft_consensus.cc:2245] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:25.403704 27637 raft_consensus.cc:2732] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.716s (election complete)
I0619 00:12:25.403832 27015 leader_election.cc:379] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.405992 27637 raft_consensus.cc:2539] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:25.407064 27637 raft_consensus.cc:593] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:25.408977 27196 ts_tablet_manager.cc:1073] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.043s	user 0.004s	sys 0.000s
I0619 00:12:25.407932 27637 consensus_queue.cc:204] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.409457 27197 raft_consensus.cc:325] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.410276 27197 raft_consensus.cc:351] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.410573 27197 raft_consensus.cc:628] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:25.411257 27197 consensus_queue.cc:227] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.414940 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.415174 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.415377 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(56d41c14ebcb4acab507b5b87f2aa8db)
I0619 00:12:25.415733 27200 raft_consensus.cc:2042] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:25.415947 27200 consensus_peers.cc:456] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:25.416213 27200 consensus_peers.cc:456] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:25.416649 27200 pending_rounds.cc:63] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Trying to abort 1 pending transactions.
I0619 00:12:25.416821 27200 pending_rounds.cc:68] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391044961689600 op_type: NO_OP noop_request { }
I0619 00:12:25.417188 27200 raft_consensus.cc:2622] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:25.417424 27200 raft_consensus.cc:2069] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.420522 27195 tablet_bootstrap.cc:427] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:25.420900 27195 ts_tablet_manager.cc:1045] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.042s	user 0.004s	sys 0.001s
I0619 00:12:25.422017 27197 ts_tablet_manager.cc:1073] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.000s
I0619 00:12:25.423411 27200 ts_tablet_manager.cc:1369] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.431013 27424 raft_consensus.cc:457] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:25.431274 27424 raft_consensus.cc:2732] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.225s (starting election)
I0619 00:12:25.431550 27424 raft_consensus.cc:479] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.432559 27424 leader_election.cc:243] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.432981 27424 leader_election.cc:243] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.434218 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e63eb8276d6d4a03b30d54126e965dbd" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:25.434667 26948 raft_consensus.cc:2732] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.207s (vote granted)
I0619 00:12:25.434913 26948 raft_consensus.cc:2245] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 0.
I0619 00:12:25.434382 26533 catalog_manager.cc:3786] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:25.435923 26732 leader_election.cc:379] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.436148 26732 leader_election.cc:270] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:25.436893 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e63eb8276d6d4a03b30d54126e965dbd" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
W0619 00:12:25.443054 26732 leader_election.cc:306] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Illegal state: must be running to vote when last-logged opid is not known
I0619 00:12:25.450935 26533 master_service.cc:523] ReplaceTablet: received request to replace tablet bef7d1a41629429bb18d6f6e859592c5 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:25.450928 27424 raft_consensus.cc:2732] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.643s (election complete)
I0619 00:12:25.451195 27424 raft_consensus.cc:2539] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:25.451398 27424 raft_consensus.cc:457] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:25.451586 27424 raft_consensus.cc:2732] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.602s (starting election)
I0619 00:12:25.451804 27424 raft_consensus.cc:2784] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.453735 27673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b46df3f26b4096876d341a82aaf7c7. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:25.454435 27673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:25.456962 27566 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" instance_seqno: 1529392334473656) as {username='jenkins'} at 127.24.119.2:33966; Asking this server to re-register.
I0619 00:12:25.458844 27567 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" instance_seqno: 1529392333071918) as {username='jenkins'} at 127.24.119.1:37713; Asking this server to re-register.
I0619 00:12:25.460023 27000 heartbeater.cc:426] Registering TS with master...
I0619 00:12:25.456332 27424 raft_consensus.cc:479] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.460582 27565 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" instance_seqno: 1529392335584735) as {username='jenkins'} at 127.24.119.3:35086; Asking this server to re-register.
I0619 00:12:25.462083 26857 heartbeater.cc:426] Registering TS with master...
I0619 00:12:25.462896 27195 raft_consensus.cc:325] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.463560 27133 heartbeater.cc:426] Registering TS with master...
I0619 00:12:25.463554 27195 raft_consensus.cc:351] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.463831 27195 raft_consensus.cc:628] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:25.464478 27195 consensus_queue.cc:227] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.465654 27565 ts_manager.cc:81] Registered new tserver with Master: 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.468432 27567 ts_manager.cc:81] Registered new tserver with Master: a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.469748 26532 master_service.cc:523] ReplaceTablet: received request to replace tablet bef7d1a41629429bb18d6f6e859592c5 from {username='jenkins'} at 127.24.119.62:33747
I0619 00:12:25.471937 27195 ts_tablet_manager.cc:1073] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.051s	user 0.004s	sys 0.001s
I0619 00:12:25.477342 27567 ts_manager.cc:81] Registered new tserver with Master: 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.479064 27424 leader_election.cc:243] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.480716 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e63eb8276d6d4a03b30d54126e965dbd" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:25.481161 26948 raft_consensus.cc:2784] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.481401 26948 raft_consensus.cc:2732] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.141s (vote granted)
I0619 00:12:25.482241 27424 leader_election.cc:243] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.483875 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e63eb8276d6d4a03b30d54126e965dbd" candidate_uuid: "10c4e713d4c44dd8904e5f77ba815785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:25.484304 27083 raft_consensus.cc:2784] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.484540 27083 raft_consensus.cc:2732] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.071s (vote granted)
I0619 00:12:25.486212 26948 raft_consensus.cc:2245] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:25.489056 26533 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet bef7d1a41629429bb18d6f6e859592c5
I0619 00:12:25.489197 26732 leader_election.cc:379] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.489434 26732 leader_election.cc:270] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:25.489439 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bef7d1a41629429bb18d6f6e859592c5 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (replaced by tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:25.490496 26792 tablet_service.cc:800] Processing DeleteTablet for tablet bef7d1a41629429bb18d6f6e859592c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:25.491091 27083 raft_consensus.cc:2245] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c4e713d4c44dd8904e5f77ba815785 in term 1.
I0619 00:12:25.491152 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bef7d1a41629429bb18d6f6e859592c5 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (replaced by tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:25.495268 26533 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet bef7d1a41629429bb18d6f6e859592c5 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (replaced by tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:25.495865 26929 tablet_service.cc:800] Processing DeleteTablet for tablet bef7d1a41629429bb18d6f6e859592c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:25.496214 27061 tablet_service.cc:800] Processing DeleteTablet for tablet bef7d1a41629429bb18d6f6e859592c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:25.496359 27210 tablet_replica.cc:261] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:25.496382 27424 raft_consensus.cc:2732] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.029s (election complete)
I0619 00:12:25.496600 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.496608 27424 raft_consensus.cc:2539] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:25.496733 27199 tablet_replica.cc:261] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:25.496814 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.496955 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.496906 27424 raft_consensus.cc:593] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Becoming Leader. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Running, Role: LEADER
I0619 00:12:25.496997 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497138 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497176 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497316 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497395 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497498 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497591 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497771 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497797 27210 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497324 27424 consensus_queue.cc:204] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.497963 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.497972 27210 raft_consensus.cc:2042] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:25.498136 27199 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.498163 27210 consensus_peers.cc:456] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:25.498329 27199 raft_consensus.cc:2042] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.498360 27210 consensus_peers.cc:456] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:25.498603 27199 raft_consensus.cc:2069] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.498735 27210 pending_rounds.cc:63] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Trying to abort 1 pending transactions.
I0619 00:12:25.498880 27210 pending_rounds.cc:68] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6264391046483582976 op_type: NO_OP noop_request { }
I0619 00:12:25.499166 27210 raft_consensus.cc:2622] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: NO_OP replication failed: Aborted: Transaction aborted
I0619 00:12:25.499382 27210 raft_consensus.cc:2069] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.500427 27199 ts_tablet_manager.cc:1369] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.501520 27210 ts_tablet_manager.cc:1369] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.514147 26533 catalog_manager.cc:4596] ReplaceTablet: tablet bef7d1a41629429bb18d6f6e859592c5 deleted and replaced by tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7
I0619 00:12:25.518165 27210 ts_tablet_manager.cc:1382] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:25.521615 26792 tablet_service.cc:759] Processing CreateTablet for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.522248 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.522716 27061 tablet_service.cc:759] Processing CreateTablet for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.523303 27061 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.520576 26532 catalog_manager.cc:3786] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1). New cstate: current_term: 1 leader_uuid: "10c4e713d4c44dd8904e5f77ba815785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:25.522655 26929 tablet_service.cc:759] Processing CreateTablet for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 (table=default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:25.525938 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:25.534251 26929 ts_tablet_manager.cc:1173] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.534631 27197 ts_tablet_manager.cc:1028] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:25.535023 27197 tablet_bootstrap.cc:427] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:25.535487 27210 log.cc:986] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/bef7d1a41629429bb18d6f6e859592c5
I0619 00:12:25.536150 27210 ts_tablet_manager.cc:1403] T bef7d1a41629429bb18d6f6e859592c5 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:25.541345 26792 ts_tablet_manager.cc:1173] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.541487 27197 tablet_bootstrap.cc:578] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:25.476270 27583 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.24.119.60:38975 (request call id 1) took 1689ms. Request Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":3720,"raft.run_wall_time_us":293849,"spinlock_wait_cycles":699572096,"child_traces":[["txn",{"apply.queue_time_us":75,"apply.run_cpu_time_us":817,"apply.run_wall_time_us":817,"mutex_wait_us":22575,"num_ops":1,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":106407,"replication_time_us":192109,"row_lock_wait_count":1,"row_lock_wait_us":105624}],["txn",{"apply.queue_time_us":66,"apply.run_cpu_time_us":895,"apply.run_wall_time_us":895,"num_ops":2,"prepare.queue_time_us":128491,"prepare.run_cpu_time_us":687,"prepare.run_wall_time_us":687,"replication_time_us":192786}],["txn",{"apply.queue_time_us":67,"apply.run_cpu_time_us":674,"apply.run_wall_time_us":673,"num_ops":1,"prepare.queue_time_us":128891,"prepare.run_cpu_time_us":520,"prepare.run_wall_time_us":64450,"replication_time_us":195607,"row_lock_wait_count":1,"row_lock_wait_us":63945}],["txn",{"apply.queue_time_us":397,"apply.run_cpu_time_us":878,"apply.run_wall_time_us":877,"num_ops":1,"prepare.queue_time_us":181667,"prepare.run_cpu_time_us":545,"prepare.run_wall_time_us":544,"replication_time_us":196716}],["txn",{"apply.queue_time_us":69,"apply.run_cpu_time_us":854,"apply.run_wall_time_us":854,"num_ops":2,"prepare.queue_time_us":175513,"prepare.run_cpu_time_us":641,"prepare.run_wall_time_us":2381,"replication_time_us":180234,"row_lock_wait_count":1,"row_lock_wait_us":1753}],["txn",{"apply.queue_time_us":67,"apply.run_cpu_time_us":668,"apply.run_wall_time_us":668,"num_ops":1,"prepare.queue_time_us":177622,"prepare.run_cpu_time_us":498,"prepare.run_wall_time_us":3150,"replication_time_us":186739,"row_lock_wait_count":1,"row_lock_wait_us":2664}],["txn",{"apply.queue_time_us":63,"apply.run_cpu_time_us":821,"apply.run_wall_time_us":819,"num_ops":2,"prepare.queue_time_us":180440,"prepare.run_cpu_time_us":585,"prepare.run_wall_time_us":584,"replication_time_us":187380}],["txn",{"apply.queue_time_us":2213,"apply.run_cpu_time_us":715,"apply.run_wall_time_us":716,"num_ops":1,"prepare.queue_time_us":180721,"prepare.run_cpu_time_us":534,"prepare.run_wall_time_us":7181,"replication_time_us":190036,"row_lock_wait_count":1,"row_lock_wait_us":6658}],["txn",{"apply.queue_time_us":2862,"apply.run_cpu_time_us":777,"apply.run_wall_time_us":775,"num_ops":2,"prepare.queue_time_us":187612,"prepare.run_cpu_time_us":902,"prepare.run_wall_time_us":902,"replication_time_us":189832,"thread_start_us":18675,"threads_started":1}],["txn",{"apply.queue_time_us":69,"apply.run_cpu_time_us":804,"apply.run_wall_time_us":5468,"num_ops":1,"prepare.queue_time_us":188227,"prepare.run_cpu_time_us":906,"prepare.run_wall_time_us":4936,"replication_time_us":208346,"row_lock_wait_count":1,"row_lock_wait_us":4041}],["txn",{"apply.queue_time_us":723,"apply.run_cpu_time_us":785,"apply.run_wall_time_us":786,"num_ops":1,"prepare.queue_time_us":192844,"prepare.run_cpu_time_us":444,"prepare.run_wall_time_us":443,"replication_time_us":208133}],["txn",{"apply.queue_time_us":2999,"apply.run_cpu_time_us":657,"apply.run_wall_time_us":658,"num_ops":1,"prepare.queue_time_us":193002,"prepare.run_cpu_time_us":444,"prepare.run_wall_time_us":445,"replication_time_us":207942,"thread_start_us":19913,"threads_started":1}],["txn",{"apply.queue_time_us":75,"apply.run_cpu_time_us":955,"apply.run_wall_time_us":955,"num_ops":2,"prepare.queue_time_us":193010,"prepare.run_cpu_time_us":516,"prepare.run_wall_time_us":516,"replication_time_us":227987}],["txn",{"apply.queue_time_us":63,"apply.run_cpu_time_us":667,"apply.run_wall_time_us":667,"num_ops":1,"prepare.queue_time_us":193188,"prepare.run_cpu_time_us":492,"prepare.run_wall_time_us":18202,"replication_time_us":229503,"row_lock_wait_count":1,"row_lock_wait_us":17723}],["txn",{"apply.queue_time_us":2725,"apply.run_cpu_time_us":687,"apply.run_wall_time_us":687,"num_ops":1,"prepare.queue_time_us":208054,"prepare.run_cpu_time_us":520,"prepare.run_wall_time_us":17235,"replication_time_us":230073,"row_lock_wait_count":1,"row_lock_wait_us":16729}],["txn",{"apply.queue_time_us":1246,"apply.run_cpu_time_us":1031,"apply.run_wall_time_us":1031,"num_ops":1,"prepare.queue_time_us":224990,"prepare.run_cpu_time_us":620,"prepare.run_wall_time_us":7000,"replication_time_us":226762,"row_lock_wait_count":1,"row_lock_wait_us":6393}],["txn",{"apply.queue_time_us":1643,"apply.run_cpu_time_us":798,"apply.run_wall_time_us":798,"num_ops":2,"prepare.queue_time_us":223376,"prepare.run_cpu_time_us":641,"prepare.run_wall_time_us":641,"replication_time_us":221534}],["txn",{"apply.queue_time_us":1618,"apply.run_cpu_time_us":622,"apply.run_wall_time_us":622,"num_ops":1,"prepare.queue_time_us":223696,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":4544,"replication_time_us":217845,"row_lock_wait_count":1,"row_lock_wait_us":3999}],["txn",{"apply.queue_time_us":1615,"apply.run_cpu_time_us":771,"apply.run_wall_time_us":771,"num_ops":2,"prepare.queue_time_us":227939,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":621,"replication_time_us":217524}],["txn",{"apply.queue_time_us":1209,"apply.run_cpu_time_us":1073,"apply.run_wall_time_us":1076,"num_ops":1,"prepare.queue_time_us":228239,"prepare.run_cpu_time_us":569,"prepare.run_wall_time_us":4291,"replication_time_us":217186,"row_lock_wait_count":1,"row_lock_wait_us":3732}],["txn",{"apply.queue_time_us":1679,"apply.run_cpu_time_us":736,"apply.run_wall_time_us":735,"num_ops":2,"prepare.queue_time_us":232246,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":628,"replication_time_us":217017}],["txn",{"apply.queue_time_us":618,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":673,"num_ops":1,"prepare.queue_time_us":232594,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":2624,"replication_time_us":216714,"row_lock_wait_count":1,"row_lock_wait_us":2085}],["txn",{"apply.queue_time_us":518,"apply.run_cpu_time_us":777,"apply.run_wall_time_us":777,"num_ops":2,"prepare.queue_time_us":234887,"prepare.run_cpu_time_us":649,"prepare.run_wall_time_us":2070,"replication_time_us":216393,"row_lock_wait_count":1,"row_lock_wait_us":785}],["txn",{"apply.queue_time_us":1181,"apply.run_cpu_time_us":627,"apply.run_wall_time_us":627,"num_ops":1,"prepare.queue_time_us":236686,"prepare.run_cpu_time_us":536,"prepare.run_wall_time_us":1415,"replication_time_us":216075,"row_lock_wait_count":1,"row_lock_wait_us":889}],["txn",{"apply.queue_time_us":1292,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":619,"num_ops":1,"prepare.queue_time_us":237732,"prepare.run_cpu_time_us":519,"prepare.run_wall_time_us":520,"replication_time_us":215780}],["txn",{"apply.queue_time_us":493,"apply.run_cpu_time_us":797,"apply.run_wall_time_us":797,"num_ops":2,"prepare.queue_time_us":237968,"prepare.run_cpu_time_us":644,"prepare.run_wall_time_us":2728,"replication_time_us":215388,"row_lock_wait_count":1,"row_lock_wait_us":2096}],["txn",{"apply.queue_time_us":1211,"apply.run_cpu_time_us":637,"apply.run_wall_time_us":7656,"num_ops":1,"prepare.queue_time_us":207674,"prepare.run_cpu_time_us":537,"prepare.run_wall_time_us":2087,"replication_time_us":182403}],["txn",{"apply.queue_time_us":6091,"apply.run_cpu_time_us":838,"apply.run_wall_time_us":1136,"num_ops":2,"prepare.queue_time_us":209412,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":620,"replication_time_us":181999}],["txn",{"num_ops":1,"prepare.queue_time_us":209707,"replication_time_us":181640,"row_lock_wait_count":1}],["txn",{"replication_time_us":181312}],["txn",{"replication_time_us":180997}],["txn",{"replication_time_us":180657}],["txn",{"replication_time_us":180280}],["txn",{"replication_time_us":179935}],["txn",{"replication_time_us":179634}],["txn",{"replication_time_us":179314}],["txn",{"replication_time_us":166179}],["txn",{"replication_time_us":165787}],["txn",{"replication_time_us":165469}],["txn",{"replication_time_us":165128}],["txn",{"replication_time_us":164800}],["txn",{"replication_time_us":164495}],["txn",{"replication_time_us":164140}],["txn",{"replication_time_us":163836}],["txn",{"replication_time_us":151295}],["txn",{"replication_time_us":149567}],["txn",{"replication_time_us":149244}],["txn",{"replication_time_us":140511}],["txn",{"replication_time_us":124550}],["txn",{"replication_time_us":124189}],["txn",{"replication_time_us":123858}],["txn",{"replication_time_us":123529}],["txn",{"replication_time_us":123215}],["txn",{"replication_time_us":122887}],["txn",{"replication_time_us":116450}],["txn",{"replication_time_us":116146}],["txn",{"replication_time_us":115836}],["txn",{"replication_time_us":115552}],["txn",{"replication_time_us":115225}],["txn",{"replication_time_us":114903}]]}
W0619 00:12:25.543525 26732 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.2:55248, user_credentials={real_user=jenkins}} blocked reactor thread for 54347us
I0619 00:12:25.544277 26732 leader_election.cc:379] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:25.544838 27196 ts_tablet_manager.cc:1028] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:25.545186 27196 tablet_bootstrap.cc:427] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:25.547410 27196 tablet_bootstrap.cc:578] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:25.547781 27061 ts_tablet_manager.cc:1173] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:25.547773 26711 sys_catalog.cc:337] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:25.548812 27195 ts_tablet_manager.cc:1028] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:25.548377 26711 sys_catalog.cc:340] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:25.551424 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet bef7d1a41629429bb18d6f6e859592c5 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:25.552886 27195 tablet_bootstrap.cc:427] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:25.555198 27195 tablet_bootstrap.cc:578] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:25.555832 27200 ts_tablet_manager.cc:1382] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:25.556284 27200 log.cc:986] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/56d41c14ebcb4acab507b5b87f2aa8db
I0619 00:12:25.556951 27200 ts_tablet_manager.cc:1403] T 56d41c14ebcb4acab507b5b87f2aa8db P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:25.558274 27637 raft_consensus.cc:457] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:25.558528 27637 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.756s (starting election)
I0619 00:12:25.559041 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 56d41c14ebcb4acab507b5b87f2aa8db (table default.table_31d388c40f064a2a97bbbec12c77fbd9 [id=16e5d101e49745c58a820c708268b84b]) successfully deleted
I0619 00:12:25.558810 27637 raft_consensus.cc:479] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.559311 27200 tablet_replica.cc:261] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:25.559538 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.559767 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.559821 27637 leader_election.cc:243] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.559959 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.560147 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.560216 27637 leader_election.cc:243] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.560372 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.561710 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "912bbbfd578c495ebd11e1eb0d170e7e" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:25.562176 26812 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.292s (vote granted)
I0619 00:12:25.562392 26812 raft_consensus.cc:2245] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:25.562691 27197 tablet_bootstrap.cc:427] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:25.563022 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "912bbbfd578c495ebd11e1eb0d170e7e" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:25.563508 26948 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.775s (vote granted)
I0619 00:12:25.563645 27199 ts_tablet_manager.cc:1382] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.563755 26948 raft_consensus.cc:2245] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:25.563889 27199 log.cc:986] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/bef7d1a41629429bb18d6f6e859592c5
I0619 00:12:25.564915 27013 leader_election.cc:379] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.565618 27013 leader_election.cc:270] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:25.566009 27637 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.820s (election complete)
I0619 00:12:25.566131 27015 leader_election.cc:379] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.566221 27637 raft_consensus.cc:2539] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:25.563005 27197 ts_tablet_manager.cc:1045] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.003s
I0619 00:12:25.566473 27637 raft_consensus.cc:457] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:25.566684 27637 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.050s (starting election)
I0619 00:12:25.566882 27637 raft_consensus.cc:2784] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.567826 27197 raft_consensus.cc:325] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.568404 27197 raft_consensus.cc:351] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.568608 27197 raft_consensus.cc:628] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:25.569111 27197 consensus_queue.cc:227] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.571123 27637 raft_consensus.cc:479] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.573536 27197 ts_tablet_manager.cc:1073] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I0619 00:12:25.574462 27637 leader_election.cc:243] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.574869 27637 leader_election.cc:243] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.575698 27195 tablet_bootstrap.cc:427] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:25.576534 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "912bbbfd578c495ebd11e1eb0d170e7e" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:25.577376 26948 raft_consensus.cc:2784] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.577569 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.577607 26948 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.772s (vote granted)
I0619 00:12:25.578001 27195 ts_tablet_manager.cc:1045] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.001s
I0619 00:12:25.578250 27200 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(bef7d1a41629429bb18d6f6e859592c5)
I0619 00:12:25.578454 27200 raft_consensus.cc:2042] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:25.575886 27199 ts_tablet_manager.cc:1403] T bef7d1a41629429bb18d6f6e859592c5 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:25.580977 27200 raft_consensus.cc:2069] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:25.581300 27195 raft_consensus.cc:325] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.582101 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet bef7d1a41629429bb18d6f6e859592c5 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:25.581951 26948 raft_consensus.cc:2245] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:25.585520 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "912bbbfd578c495ebd11e1eb0d170e7e" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:25.585988 26812 raft_consensus.cc:2784] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.586243 26812 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.793s (vote granted)
I0619 00:12:25.589812 27196 tablet_bootstrap.cc:427] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:25.590185 27196 ts_tablet_manager.cc:1045] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.045s	user 0.004s	sys 0.004s
I0619 00:12:25.583876 27013 leader_election.cc:379] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.590430 27013 leader_election.cc:270] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:25.590426 26812 raft_consensus.cc:2245] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:25.590977 27195 raft_consensus.cc:351] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.591197 27195 raft_consensus.cc:628] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:25.591452 27015 leader_election.cc:379] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.591725 27637 raft_consensus.cc:2732] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.013s (election complete)
I0619 00:12:25.591504 27196 raft_consensus.cc:325] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.591976 27637 raft_consensus.cc:2539] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:25.592124 27196 raft_consensus.cc:351] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:25.592278 27637 raft_consensus.cc:593] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:25.592319 27196 raft_consensus.cc:628] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:25.592208 27195 consensus_queue.cc:227] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.592773 27196 consensus_queue.cc:227] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.584414 27200 ts_tablet_manager.cc:1369] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:25.597097 27196 ts_tablet_manager.cc:1073] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I0619 00:12:25.592819 27637 consensus_queue.cc:204] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.606907 27195 ts_tablet_manager.cc:1073] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.000s
I0619 00:12:25.627063 26532 catalog_manager.cc:3786] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:25.638907 25052 master-stress-test.cc:468] Killing test thread 1/10
I0619 00:12:25.657039 27200 ts_tablet_manager.cc:1382] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I0619 00:12:25.657384 27200 log.cc:986] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/bef7d1a41629429bb18d6f6e859592c5
I0619 00:12:25.658093 27200 ts_tablet_manager.cc:1403] T bef7d1a41629429bb18d6f6e859592c5 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:25.659915 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet bef7d1a41629429bb18d6f6e859592c5 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:25.688665 27637 raft_consensus.cc:457] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:25.688926 27637 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.099s (starting election)
I0619 00:12:25.689191 27637 raft_consensus.cc:479] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.690258 27637 leader_election.cc:243] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.690693 27637 leader_election.cc:243] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.692494 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "55aa16ffaca64a1ca2dc2e2cb236cfd7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:25.692519 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "55aa16ffaca64a1ca2dc2e2cb236cfd7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:25.692958 26948 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.085s (vote granted)
I0619 00:12:25.693006 26812 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.666s (vote granted)
I0619 00:12:25.693168 26948 raft_consensus.cc:2245] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:25.693228 26812 raft_consensus.cc:2245] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:25.694449 27013 leader_election.cc:379] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.694705 27013 leader_election.cc:270] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:25.694948 27015 leader_election.cc:379] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.695179 27637 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.567s (election complete)
I0619 00:12:25.695446 27637 raft_consensus.cc:2539] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:25.695678 27637 raft_consensus.cc:457] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:25.695881 27637 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.816s (starting election)
I0619 00:12:25.696082 27637 raft_consensus.cc:2784] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.700903 27637 raft_consensus.cc:479] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.701813 27637 leader_election.cc:243] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.702190 27637 leader_election.cc:243] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.703804 26948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "55aa16ffaca64a1ca2dc2e2cb236cfd7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:25.703814 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "55aa16ffaca64a1ca2dc2e2cb236cfd7" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:25.704216 26948 raft_consensus.cc:2784] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.704218 26812 raft_consensus.cc:2784] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:25.704452 26948 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.696s (vote granted)
I0619 00:12:25.704452 26812 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.750s (vote granted)
I0619 00:12:25.708884 26812 raft_consensus.cc:2245] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:25.709400 26948 raft_consensus.cc:2245] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:25.710083 27015 leader_election.cc:379] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:25.710316 27015 leader_election.cc:270] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:25.710896 27546 raft_consensus.cc:2732] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.004s (election complete)
I0619 00:12:25.711133 27546 raft_consensus.cc:2539] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:25.712236 27013 leader_election.cc:379] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:25.719331 27546 raft_consensus.cc:593] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:25.719832 27546 consensus_queue.cc:204] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:25.738643 26532 catalog_manager.cc:3786] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:25.815668 26948 raft_consensus.cc:1081] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:25.817001 27546 consensus_queue.cc:963] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:25.837321 26812 raft_consensus.cc:1081] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:25.842346 27710 consensus_queue.cc:963] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:25.917114 27083 raft_consensus.cc:1081] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:25.918535 27703 consensus_queue.cc:963] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:25.934264 26948 raft_consensus.cc:1081] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer 10c4e713d4c44dd8904e5f77ba815785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:25.938513 27703 consensus_queue.cc:963] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:26.088423 26812 raft_consensus.cc:1081] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:26.090037 27710 consensus_queue.cc:963] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:26.114950 26948 raft_consensus.cc:1081] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:26.118103 27710 consensus_queue.cc:963] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:26.143498 26948 raft_consensus.cc:1081] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:26.145081 27710 consensus_queue.cc:963] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:26.161185 26811 raft_consensus.cc:1081] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:26.165309 27546 consensus_queue.cc:963] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0619 00:12:26.171154 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:26.263300 27721 raft_consensus.cc:457] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:26.263661 27721 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.704s (starting election)
I0619 00:12:26.263999 27721 raft_consensus.cc:479] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:26.268987 27721 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:26.269603 27721 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:26.273943 27689 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:26.297514 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 178 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:26.298122 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:26.299152 27525 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:26.324584 27341 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 178 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
I0619 00:12:26.325290 27341 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:26.326172 27545 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000). Message: Invalid argument: T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:26.326439 27545 leader_election.cc:270] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:26.327002 27721 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.734s (election complete)
I0619 00:12:26.327215 27721 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:26.347100 27691 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: acquiring CA information for follower catalog manager: success
I0619 00:12:26.366569 27691 catalog_manager.cc:1090] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W0619 00:12:26.482549 26539 rpcz_store.cc:251] Call kudu.master.MasterService.AlterTable from 127.24.119.62:33747 (request call id 6) took 9999ms (client timeout 9999).
W0619 00:12:26.483484 26539 rpcz_store.cc:255] Trace:
0619 00:12:16.483128 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:16.483272 (+   144us) service_pool.cc:221] Handling call
0619 00:12:17.285356 (+802084us) catalog_manager.cc:1680] Looking up and locking table
0619 00:12:26.482471 (+9197115us) inbound_call.cc:162] Queueing success response
Metrics: {"hms-catalog.queue_time_us":4835,"hms-catalog.run_cpu_time_us":1051,"hms-catalog.run_wall_time_us":1155505}
W0619 00:12:26.484688 27151 client-internal.cc:262] Unable to send the request (table { table_name: "default.table_4a6e4e28e18547719a48db319fc95750" } new_table_name: "default.table_8a99507c6bce41038f15f6aea10b464e" alter_external_catalogs: true) to leader Master (127.24.119.62:58195): Timed out: AlterTable RPC to 127.24.119.62:58195 timed out after 10.000s (SENT)
I0619 00:12:26.500236 27151 client-internal.cc:267] Determining the new leader Master and retrying...
I0619 00:12:26.673141 27152 client-internal.cc:267] Determining the new leader Master and retrying...
W0619 00:12:26.684490 27149 client-internal.cc:262] Unable to send the request (name: "default.table_f4f93d7c11cf4735980b09abd2aa7f62" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } }) to leader Master (127.24.119.62:58195): Timed out: CreateTable RPC to 127.24.119.62:58195 timed out after 10.000s (SENT)
I0619 00:12:26.693398 27149 client-internal.cc:267] Determining the new leader Master and retrying...
I0619 00:12:26.875725 27147 client-internal.cc:267] Determining the new leader Master and retrying...
W0619 00:12:26.960302 27153 client-internal.cc:262] Unable to send the request (table { table_name: "default.table_f9c29cfbe84f43ff836467ab27c1a7ca" }) to leader Master (127.24.119.62:58195): Timed out: DeleteTable RPC to 127.24.119.62:58195 timed out after 10.000s (SENT)
I0619 00:12:26.960661 27153 client-internal.cc:267] Determining the new leader Master and retrying...
I0619 00:12:27.162461 27148 client-internal.cc:267] Determining the new leader Master and retrying...
I0619 00:12:27.173111 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:27.224838 27154 client-internal.cc:267] Determining the new leader Master and retrying...
I0619 00:12:27.274955 27689 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
W0619 00:12:27.422529 26534 rpcz_store.cc:251] Call kudu.master.MasterService.CreateTable from 127.24.119.62:33747 (request call id 21) took 9999ms (client timeout 9999).
W0619 00:12:27.423099 26534 rpcz_store.cc:255] Trace:
0619 00:12:17.423203 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:17.423296 (+    93us) service_pool.cc:221] Handling call
0619 00:12:27.422466 (+9999170us) inbound_call.cc:162] Queueing success response
Metrics: {}
I0619 00:12:27.425562 27150 client-internal.cc:267] Determining the new leader Master and retrying...
W0619 00:12:27.470868 26712 hms_client.cc:296] Time spent drop HMS table: real 9.025s	user 0.000s	sys 0.001s
W0619 00:12:27.471505 26538 rpcz_store.cc:251] Call kudu.master.MasterService.DeleteTable from 127.24.119.62:33747 (request call id 18) took 10247ms (client timeout 9999).
W0619 00:12:27.472236 26538 rpcz_store.cc:255] Trace:
0619 00:12:17.224261 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:17.224732 (+   471us) service_pool.cc:221] Handling call
0619 00:12:17.290865 (+ 66133us) catalog_manager.cc:1680] Looking up and locking table
0619 00:12:27.471462 (+10180597us) inbound_call.cc:162] Queueing success response
Metrics: {"hms-catalog.queue_time_us":1154950,"hms-catalog.run_cpu_time_us":983,"hms-catalog.run_wall_time_us":9025084}
I0619 00:12:27.472839 26538 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:33747:
table { table_name: "default.table_d9d85b384d834dd9ac9814f9d0fa9ce3" }
W0619 00:12:27.595271 26541 rpcz_store.cc:251] Call kudu.master.MasterService.AlterTable from 127.24.119.62:33747 (request call id 9) took 10923ms (client timeout 9999).
W0619 00:12:27.596053 26541 rpcz_store.cc:255] Trace:
0619 00:12:16.671620 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:16.672228 (+   608us) service_pool.cc:221] Handling call
0619 00:12:17.294480 (+622252us) catalog_manager.cc:1680] Looking up and locking table
0619 00:12:27.595213 (+10300733us) inbound_call.cc:162] Queueing success response
Metrics: {"hms-catalog.queue_time_us":10176506,"hms-catalog.run_cpu_time_us":1073,"hms-catalog.run_wall_time_us":123669}
I0619 00:12:27.639865 27732 raft_consensus.cc:457] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:27.640223 27732 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.820s (starting election)
I0619 00:12:27.640523 27732 raft_consensus.cc:479] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:27.646582 27732 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:27.647451 27732 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:27.648264 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 178 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:27.648851 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.649554 27253 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.682238 27582 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 178 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696" is_pre_election: true
I0619 00:12:27.682955 27582 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.683745 27254 leader_election.cc:393] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918). Message: Invalid argument: T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8582cfc277d4d4597c5605b7ac0ed48 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.684032 27254 leader_election.cc:270] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:27.684501 27732 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.749s (election complete)
I0619 00:12:27.684744 27732 raft_consensus.cc:2487] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W0619 00:12:27.789631 26540 rpcz_store.cc:251] Call kudu.master.MasterService.DeleteTable from 127.24.119.62:33747 (request call id 14) took 10830ms (client timeout 9999).
W0619 00:12:27.790364 26540 rpcz_store.cc:255] Trace:
0619 00:12:16.959325 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:16.961859 (+  2534us) service_pool.cc:221] Handling call
0619 00:12:17.294747 (+332888us) catalog_manager.cc:1680] Looking up and locking table
0619 00:12:27.789550 (+10494803us) inbound_call.cc:162] Queueing success response
Metrics: {"hms-catalog.queue_time_us":10300047,"hms-catalog.run_cpu_time_us":286,"hms-catalog.run_wall_time_us":194252}
I0619 00:12:27.811341 27737 raft_consensus.cc:457] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:27.811638 27737 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.605s (starting election)
I0619 00:12:27.811967 27737 raft_consensus.cc:479] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:27.813325 27737 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:27.813726 27737 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:27.814539 26551 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 178 } } ignore_live_leader: false dest_uuid: "90b46df3f26b4096876d341a82aaf7c7" is_pre_election: true
I0619 00:12:27.814793 27341 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 178 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
I0619 00:12:27.815116 26551 raft_consensus.cc:2206] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.815310 27341 raft_consensus.cc:2206] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.815778 27525 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195). Message: Invalid argument: T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.820530 27545 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000). Message: Invalid argument: T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 for term 2 because replica is either leader or believes a valid leader to be alive.
I0619 00:12:27.821202 27545 leader_election.cc:270] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I0619 00:12:27.825206 27737 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.740s (election complete)
I0619 00:12:27.825450 27737 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W0619 00:12:27.921618 26536 rpcz_store.cc:251] Call kudu.master.MasterService.CreateTable from 127.24.119.62:33747 (request call id 17) took 10754ms (client timeout 9999).
W0619 00:12:27.924048 26536 rpcz_store.cc:255] Trace:
0619 00:12:17.166904 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:17.167770 (+   866us) service_pool.cc:221] Handling call
0619 00:12:17.292166 (+124396us) catalog_manager.cc:1494] Acquired catalog manager lock
0619 00:12:17.295435 (+  3269us) catalog_manager.cc:1538] Created new table and tablet info
0619 00:12:27.921192 (+10625757us) catalog_manager.cc:1584] Wrote table and tablets to system table
0619 00:12:27.921359 (+   167us) catalog_manager.cc:1618] Inserted table and tablets into CatalogManager maps
0619 00:12:27.921566 (+   207us) inbound_call.cc:162] Queueing success response
Related trace 'txn':
0619 00:12:27.877419 (+     0us) write_transaction.cc:100] PREPARE: Starting
0619 00:12:27.877558 (+   139us) write_transaction.cc:267] Acquiring schema lock in shared mode
0619 00:12:27.877580 (+    22us) write_transaction.cc:270] Acquired schema lock
0619 00:12:27.877595 (+    15us) tablet.cc:399] PREPARE: Decoding operations
0619 00:12:27.877782 (+   187us) tablet.cc:421] PREPARE: Acquiring locks for 2 operations
0619 00:12:27.878003 (+   221us) tablet.cc:425] PREPARE: locks acquired
0619 00:12:27.878020 (+    17us) write_transaction.cc:125] PREPARE: finished.
0619 00:12:27.878094 (+    74us) write_transaction.cc:135] Start()
0619 00:12:27.878160 (+    66us) write_transaction.cc:140] Timestamp: P: 1529392347878069 usec, L: 0
0619 00:12:27.878958 (+   798us) log.cc:587] Serialized 410 byte log entry
0619 00:12:27.917950 (+ 38992us) write_transaction.cc:148] APPLY: Starting
0619 00:12:27.918433 (+   483us) tablet_metrics.cc:365] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0619 00:12:27.918623 (+   190us) log.cc:587] Serialized 29 byte log entry
0619 00:12:27.920403 (+  1780us) write_transaction.cc:308] Releasing row and schema locks
0619 00:12:27.920462 (+    59us) write_transaction.cc:276] Released schema lock
0619 00:12:27.920496 (+    34us) write_transaction.cc:195] FINISH: updating metrics
Metrics: {"hms-catalog.queue_time_us":10493792,"hms-catalog.run_cpu_time_us":572,"hms-catalog.run_wall_time_us":75599,"child_traces":[["txn",{"apply.queue_time_us":10144,"num_ops":2,"prepare.queue_time_us":11662,"prepare.run_cpu_time_us":4985,"prepare.run_wall_time_us":9791,"raft.queue_time_us":150,"raft.run_cpu_time_us":1142,"raft.run_wall_time_us":1173,"replication_time_us":29463,"spinlock_wait_cycles":545024,"thread_start_us":24482,"threads_started":3,"wal-append.queue_time_us":5843}]]}
I0619 00:12:27.924527 26536 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.62:33747:
table { table_name: "default.table_f9c29cfbe84f43ff836467ab27c1a7ca" }
I0619 00:12:27.939244 26929 tablet_service.cc:759] Processing CreateTablet for tablet ca7268d23f314af5a87b71a79246c324 (table=default.table_3a31a472b2e1460591b55ec649f29582 [id=5c424f62ca27467eb104ac30e5e4ae56]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:27.939954 26929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ca7268d23f314af5a87b71a79246c324. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:27.939981 27061 tablet_service.cc:759] Processing CreateTablet for tablet ca7268d23f314af5a87b71a79246c324 (table=default.table_3a31a472b2e1460591b55ec649f29582 [id=5c424f62ca27467eb104ac30e5e4ae56]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:27.940575 27061 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ca7268d23f314af5a87b71a79246c324. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:27.943823 26792 tablet_service.cc:759] Processing CreateTablet for tablet ca7268d23f314af5a87b71a79246c324 (table=default.table_3a31a472b2e1460591b55ec649f29582 [id=5c424f62ca27467eb104ac30e5e4ae56]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:27.944411 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ca7268d23f314af5a87b71a79246c324. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:27.949774 26929 ts_tablet_manager.cc:1173] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:27.950726 27061 ts_tablet_manager.cc:1173] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:27.953248 26792 ts_tablet_manager.cc:1173] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:27.962774 26537 rpcz_store.cc:251] Call kudu.master.MasterService.CreateTable from 127.24.119.62:33747 (request call id 10) took 11278ms (client timeout 9999).
W0619 00:12:27.965343 26537 rpcz_store.cc:255] Trace:
0619 00:12:16.684447 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:16.684535 (+    88us) service_pool.cc:221] Handling call
0619 00:12:17.294199 (+609664us) catalog_manager.cc:1494] Acquired catalog manager lock
0619 00:12:17.296021 (+  1822us) catalog_manager.cc:1538] Created new table and tablet info
0619 00:12:27.962326 (+10666305us) catalog_manager.cc:1584] Wrote table and tablets to system table
0619 00:12:27.962505 (+   179us) catalog_manager.cc:1618] Inserted table and tablets into CatalogManager maps
0619 00:12:27.962722 (+   217us) inbound_call.cc:162] Queueing success response
Related trace 'txn':
0619 00:12:27.951696 (+     0us) spinlock_profiling.cc:243] Waited 9.9 ms on lock 0x7b4c0001055c. stack: 00007ffb75706614 00007ffb75706510 00007ffb750e01b4 00007ffb7dddc074 00007ffb7dddc009 00007ffb78efb6a6 00007ffb7bd3d373 00007ffb7bd2ee14 00007ffb7de1ba27 00007ffb7de1bf1a 00007ffb7dd9c616 00007ffb7de1249f 00007ffb7c6dafe4 00007ffb7c6daf71 00007ffb7c6dae84 00007ffb76f85ce9
0619 00:12:27.951843 (+   147us) write_transaction.cc:100] PREPARE: Starting
0619 00:12:27.951981 (+   138us) write_transaction.cc:267] Acquiring schema lock in shared mode
0619 00:12:27.952002 (+    21us) write_transaction.cc:270] Acquired schema lock
0619 00:12:27.952015 (+    13us) tablet.cc:399] PREPARE: Decoding operations
0619 00:12:27.952177 (+   162us) tablet.cc:421] PREPARE: Acquiring locks for 2 operations
0619 00:12:27.952397 (+   220us) tablet.cc:425] PREPARE: locks acquired
0619 00:12:27.952412 (+    15us) write_transaction.cc:125] PREPARE: finished.
0619 00:12:27.952480 (+    68us) write_transaction.cc:135] Start()
0619 00:12:27.952545 (+    65us) write_transaction.cc:140] Timestamp: P: 1529392347952457 usec, L: 0
0619 00:12:27.952929 (+   384us) log.cc:587] Serialized 410 byte log entry
0619 00:12:27.960364 (+  7435us) write_transaction.cc:148] APPLY: Starting
0619 00:12:27.960859 (+   495us) tablet_metrics.cc:365] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0619 00:12:27.961028 (+   169us) log.cc:587] Serialized 29 byte log entry
0619 00:12:27.961114 (+    86us) write_transaction.cc:308] Releasing row and schema locks
0619 00:12:27.961166 (+    52us) write_transaction.cc:276] Released schema lock
0619 00:12:27.962198 (+  1032us) write_transaction.cc:195] FINISH: updating metrics
Metrics: {"hms-catalog.queue_time_us":10568895,"hms-catalog.run_cpu_time_us":502,"hms-catalog.run_wall_time_us":75227,"child_traces":[["txn",{"apply.queue_time_us":1609,"num_ops":2,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":1321,"prepare.run_wall_time_us":1321,"raft.queue_time_us":1011,"raft.run_cpu_time_us":789,"raft.run_wall_time_us":7406,"replication_time_us":6000,"spinlock_wait_cycles":23768192}]]}
I0619 00:12:27.966948 27762 ts_tablet_manager.cc:1028] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:27.967203 27763 ts_tablet_manager.cc:1028] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:27.971670 27762 tablet_bootstrap.cc:427] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:27.972069 27763 tablet_bootstrap.cc:427] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:27.975342 26928 tablet_service.cc:759] Processing CreateTablet for tablet 954b5c3ea0d44539b2466a5fba4df731 (table=default.table_f4f93d7c11cf4735980b09abd2aa7f62 [id=602806a614894f7c8b6d41137a33dcd5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:27.975842 26792 tablet_service.cc:759] Processing CreateTablet for tablet 954b5c3ea0d44539b2466a5fba4df731 (table=default.table_f4f93d7c11cf4735980b09abd2aa7f62 [id=602806a614894f7c8b6d41137a33dcd5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:27.975997 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 954b5c3ea0d44539b2466a5fba4df731. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:27.976420 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 954b5c3ea0d44539b2466a5fba4df731. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:27.976624 27061 tablet_service.cc:759] Processing CreateTablet for tablet 954b5c3ea0d44539b2466a5fba4df731 (table=default.table_f4f93d7c11cf4735980b09abd2aa7f62 [id=602806a614894f7c8b6d41137a33dcd5]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:27.977236 27061 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 954b5c3ea0d44539b2466a5fba4df731. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:27.984762 27763 tablet_bootstrap.cc:578] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:27.987639 26792 ts_tablet_manager.cc:1173] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:27.987774 27761 ts_tablet_manager.cc:1028] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:27.988317 27761 tablet_bootstrap.cc:427] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:27.988788 26928 ts_tablet_manager.cc:1173] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:27.990345 27061 ts_tablet_manager.cc:1173] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:27.990767 27761 tablet_bootstrap.cc:578] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:27.993719 27761 tablet_bootstrap.cc:427] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:27.993929 27761 ts_tablet_manager.cc:1045] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.004s
I0619 00:12:27.994915 27761 raft_consensus.cc:325] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:27.995306 27761 raft_consensus.cc:351] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:27.995476 27761 raft_consensus.cc:628] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:27.996507 27762 tablet_bootstrap.cc:578] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:27.996513 27761 consensus_queue.cc:227] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:27.999454 27763 tablet_bootstrap.cc:427] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:27.999786 27763 ts_tablet_manager.cc:1045] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.028s	user 0.003s	sys 0.001s
I0619 00:12:28.001207 27763 raft_consensus.cc:325] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.004096 27763 raft_consensus.cc:351] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.004302 27763 raft_consensus.cc:628] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:28.005311 27762 tablet_bootstrap.cc:427] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:28.005642 27762 ts_tablet_manager.cc:1045] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.034s	user 0.005s	sys 0.000s
I0619 00:12:28.006922 27762 raft_consensus.cc:325] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.007526 27762 raft_consensus.cc:351] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.007751 27762 raft_consensus.cc:628] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:28.008190 27762 consensus_queue.cc:227] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.009212 27763 consensus_queue.cc:227] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.015302 27762 ts_tablet_manager.cc:1073] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I0619 00:12:28.017200 27763 ts_tablet_manager.cc:1073] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.000s
I0619 00:12:28.018808 27763 ts_tablet_manager.cc:1028] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:28.018996 27762 ts_tablet_manager.cc:1028] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:28.019356 27762 tablet_bootstrap.cc:427] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:28.019734 27763 tablet_bootstrap.cc:427] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:28.026923 27762 tablet_bootstrap.cc:578] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:28.029155 27761 ts_tablet_manager.cc:1073] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.000s
I0619 00:12:28.030164 27763 tablet_bootstrap.cc:578] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:28.030514 27761 ts_tablet_manager.cc:1028] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:28.030937 27761 tablet_bootstrap.cc:427] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:28.033668 27761 tablet_bootstrap.cc:578] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:28.048300 27761 tablet_bootstrap.cc:427] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:28.048662 27761 ts_tablet_manager.cc:1045] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.012s
I0619 00:12:28.049902 27761 raft_consensus.cc:325] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.050305 27761 raft_consensus.cc:351] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.050446 27761 raft_consensus.cc:628] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
I0619 00:12:28.050822 27761 consensus_queue.cc:227] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.052134 27762 tablet_bootstrap.cc:427] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:28.052429 27762 ts_tablet_manager.cc:1045] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.033s	user 0.004s	sys 0.005s
I0619 00:12:28.053727 27762 raft_consensus.cc:325] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.054333 27762 raft_consensus.cc:351] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.054546 27762 raft_consensus.cc:628] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:28.055670 27763 tablet_bootstrap.cc:427] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:28.056015 27763 ts_tablet_manager.cc:1045] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.036s	user 0.003s	sys 0.006s
I0619 00:12:28.057332 27763 raft_consensus.cc:325] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.057924 27763 raft_consensus.cc:351] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.058112 27763 raft_consensus.cc:628] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:28.060005 27762 consensus_queue.cc:227] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.058547 27763 consensus_queue.cc:227] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.062458 27763 ts_tablet_manager.cc:1073] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I0619 00:12:28.064644 27762 ts_tablet_manager.cc:1073] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.001s
I0619 00:12:28.077831 27761 ts_tablet_manager.cc:1073] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.029s	user 0.003s	sys 0.000s
W0619 00:12:28.094254 26535 rpcz_store.cc:251] Call kudu.master.MasterService.CreateTable from 127.24.119.62:33747 (request call id 13) took 11219ms (client timeout 9999).
W0619 00:12:28.097194 26535 rpcz_store.cc:255] Trace:
0619 00:12:16.875106 (+     0us) service_pool.cc:162] Inserting onto call queue
0619 00:12:16.875222 (+   116us) service_pool.cc:221] Handling call
0619 00:12:17.291931 (+416709us) catalog_manager.cc:1494] Acquired catalog manager lock
0619 00:12:17.297022 (+  5091us) catalog_manager.cc:1538] Created new table and tablet info
0619 00:12:28.093869 (+10796847us) catalog_manager.cc:1584] Wrote table and tablets to system table
0619 00:12:28.094035 (+   166us) catalog_manager.cc:1618] Inserted table and tablets into CatalogManager maps
0619 00:12:28.094203 (+   168us) inbound_call.cc:162] Queueing success response
Related trace 'txn':
0619 00:12:28.086557 (+     0us) write_transaction.cc:100] PREPARE: Starting
0619 00:12:28.086721 (+   164us) write_transaction.cc:267] Acquiring schema lock in shared mode
0619 00:12:28.086742 (+    21us) write_transaction.cc:270] Acquired schema lock
0619 00:12:28.086756 (+    14us) tablet.cc:399] PREPARE: Decoding operations
0619 00:12:28.086914 (+   158us) tablet.cc:421] PREPARE: Acquiring locks for 2 operations
0619 00:12:28.087161 (+   247us) tablet.cc:425] PREPARE: locks acquired
0619 00:12:28.087176 (+    15us) write_transaction.cc:125] PREPARE: finished.
0619 00:12:28.087252 (+    76us) write_transaction.cc:135] Start()
0619 00:12:28.087315 (+    63us) write_transaction.cc:140] Timestamp: P: 1529392348087224 usec, L: 0
0619 00:12:28.087674 (+   359us) log.cc:587] Serialized 410 byte log entry
0619 00:12:28.092741 (+  5067us) write_transaction.cc:148] APPLY: Starting
0619 00:12:28.093158 (+   417us) tablet_metrics.cc:365] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0619 00:12:28.093330 (+   172us) log.cc:587] Serialized 29 byte log entry
0619 00:12:28.093407 (+    77us) write_transaction.cc:308] Releasing row and schema locks
0619 00:12:28.093457 (+    50us) write_transaction.cc:276] Released schema lock
0619 00:12:28.093490 (+    33us) write_transaction.cc:195] FINISH: updating metrics
0619 00:12:28.093661 (+   171us) write_transaction.cc:308] Releasing row and schema locks
0619 00:12:28.093682 (+    21us) write_transaction.cc:276] Released schema lock
Metrics: {"hms-catalog.queue_time_us":10643190,"hms-catalog.run_cpu_time_us":513,"hms-catalog.run_wall_time_us":145337,"child_traces":[["txn",{"apply.queue_time_us":170,"apply.run_cpu_time_us":1040,"apply.run_wall_time_us":1040,"num_ops":2,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":1379,"prepare.run_wall_time_us":1379,"raft.queue_time_us":576,"raft.run_cpu_time_us":842,"raft.run_wall_time_us":843,"replication_time_us":5118}]]}
I0619 00:12:28.110798 26792 tablet_service.cc:759] Processing CreateTablet for tablet 65392b375030460abad2b8dcd751a238 (table=default.table_06c34c9cf69d48a7be4493acace87da2 [id=9354ca9558ad49bb92a441c542c973cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:28.111380 26792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 65392b375030460abad2b8dcd751a238. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:28.112759 27061 tablet_service.cc:759] Processing CreateTablet for tablet 65392b375030460abad2b8dcd751a238 (table=default.table_06c34c9cf69d48a7be4493acace87da2 [id=9354ca9558ad49bb92a441c542c973cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:28.113057 26928 tablet_service.cc:759] Processing CreateTablet for tablet 65392b375030460abad2b8dcd751a238 (table=default.table_06c34c9cf69d48a7be4493acace87da2 [id=9354ca9558ad49bb92a441c542c973cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:28.113358 27061 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 65392b375030460abad2b8dcd751a238. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:28.113683 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 65392b375030460abad2b8dcd751a238. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:28.123070 26928 ts_tablet_manager.cc:1173] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:28.123430 27761 ts_tablet_manager.cc:1028] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrapping tablet
I0619 00:12:28.123821 27761 tablet_bootstrap.cc:427] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Bootstrap starting.
I0619 00:12:28.124760 26792 ts_tablet_manager.cc:1173] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:28.126319 27761 tablet_bootstrap.cc:578] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: No blocks or log segments found. Creating new log.
I0619 00:12:28.127873 27061 ts_tablet_manager.cc:1173] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:28.127987 27762 ts_tablet_manager.cc:1028] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrapping tablet
I0619 00:12:28.128322 27762 tablet_bootstrap.cc:427] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: Bootstrap starting.
I0619 00:12:28.130370 27762 tablet_bootstrap.cc:578] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: No blocks or log segments found. Creating new log.
I0619 00:12:28.131496 26713 catalog_manager.cc:1776] Deleting table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f] in response to Hive Metastore notification log event 6
I0619 00:12:28.133997 27763 ts_tablet_manager.cc:1028] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:28.134347 27763 tablet_bootstrap.cc:427] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:28.136638 27763 tablet_bootstrap.cc:578] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:28.137766 27771 raft_consensus.cc:457] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:28.138039 27771 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.630s (starting election)
I0619 00:12:28.138276 27771 raft_consensus.cc:479] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.139284 27771 leader_election.cc:243] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.139719 27771 leader_election.cc:243] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.141449 27762 tablet_bootstrap.cc:427] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: No bootstrap required, opened a new log
I0619 00:12:28.141196 26811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca7268d23f314af5a87b71a79246c324" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:28.141836 27763 tablet_bootstrap.cc:427] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:28.141762 26811 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.628s (vote granted)
I0619 00:12:28.142105 27763 ts_tablet_manager.cc:1045] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.001s
I0619 00:12:28.142031 26811 raft_consensus.cc:2245] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:28.143266 26876 leader_election.cc:379] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.143517 26876 leader_election.cc:270] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:28.143820 27763 raft_consensus.cc:325] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.144407 27763 raft_consensus.cc:351] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.144455 27771 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.737s (election complete)
I0619 00:12:28.144735 27771 raft_consensus.cc:2539] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:28.144933 27771 raft_consensus.cc:457] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:28.145124 27771 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 2.247s (starting election)
I0619 00:12:28.145319 27771 raft_consensus.cc:2784] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.144582 27763 raft_consensus.cc:628] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:28.146170 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca7268d23f314af5a87b71a79246c324" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:28.146066 27763 consensus_queue.cc:227] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.146622 27083 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 2.096s (vote granted)
I0619 00:12:28.146873 27083 raft_consensus.cc:2245] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:28.148344 26874 leader_election.cc:379] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.152014 27763 ts_tablet_manager.cc:1073] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I0619 00:12:28.141779 27762 ts_tablet_manager.cc:1045] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.002s
I0619 00:12:28.154428 27762 raft_consensus.cc:325] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.155115 27762 raft_consensus.cc:351] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.154904 27771 raft_consensus.cc:479] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.155328 27762 raft_consensus.cc:628] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c4e713d4c44dd8904e5f77ba815785, State: Initialized, Role: FOLLOWER
I0619 00:12:28.155911 27771 leader_election.cc:243] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.156294 27771 leader_election.cc:243] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.155956 27762 consensus_queue.cc:227] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.156898 26713 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet e63eb8276d6d4a03b30d54126e965dbd
I0619 00:12:28.157186 26713 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e63eb8276d6d4a03b30d54126e965dbd on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (Table deleted at 2018-06-19 00:12:28 PDT)
I0619 00:12:28.159029 26713 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e63eb8276d6d4a03b30d54126e965dbd on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (Table deleted at 2018-06-19 00:12:28 PDT)
I0619 00:12:28.159694 26713 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e63eb8276d6d4a03b30d54126e965dbd on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (Table deleted at 2018-06-19 00:12:28 PDT)
I0619 00:12:28.160573 27061 tablet_service.cc:800] Processing DeleteTablet for tablet e63eb8276d6d4a03b30d54126e965dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:28 PDT) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:28.160619 26928 tablet_service.cc:800] Processing DeleteTablet for tablet e63eb8276d6d4a03b30d54126e965dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:28 PDT) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:28.163496 27762 ts_tablet_manager.cc:1073] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I0619 00:12:28.165935 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca7268d23f314af5a87b71a79246c324" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:28.166378 27083 raft_consensus.cc:2784] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.166617 27083 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.563s (vote granted)
I0619 00:12:28.169462 26792 tablet_service.cc:800] Processing DeleteTablet for tablet e63eb8276d6d4a03b30d54126e965dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:28 PDT) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:28.170904 27083 raft_consensus.cc:2245] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:28.171886 26874 leader_election.cc:379] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.172098 26874 leader_election.cc:270] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:28.174481 27438 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: b8582cfc277d4d4597c5605b7ac0ed48, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:28.176995 27783 tablet_replica.cc:261] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:28.177294 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.177523 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.179065 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.172509 27771 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.871s (election complete)
I0619 00:12:28.179596 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.179855 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.180078 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.180294 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.184803 27637 raft_consensus.cc:457] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:28.188158 27637 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.650s (starting election)
I0619 00:12:28.188161 27783 raft_consensus.cc:2042] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:28.188673 27783 raft_consensus.cc:2069] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:28.188860 27785 tablet_replica.cc:261] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:28.188493 27637 raft_consensus.cc:479] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.189088 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.189267 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.189440 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.189512 27637 leader_election.cc:243] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:28.189646 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.189847 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.189929 27637 leader_election.cc:243] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.190032 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.190198 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.190363 27785 raft_consensus.cc:2042] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:28.192243 27771 raft_consensus.cc:2539] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:28.192576 27771 raft_consensus.cc:593] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:28.191725 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "954b5c3ea0d44539b2466a5fba4df731" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:28.193347 26812 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.778s (vote granted)
I0619 00:12:28.193048 27771 consensus_queue.cc:204] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.193565 26812 raft_consensus.cc:2245] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:28.192999 27761 tablet_bootstrap.cc:427] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: No bootstrap required, opened a new log
I0619 00:12:28.194137 27761 ts_tablet_manager.cc:1045] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Time spent bootstrapping tablet: real 0.070s	user 0.004s	sys 0.010s
I0619 00:12:28.195144 27015 leader_election.cc:379] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.195379 27015 leader_election.cc:270] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:28.195860 27637 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.853s (election complete)
I0619 00:12:28.196082 27637 raft_consensus.cc:2539] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:28.196302 27637 raft_consensus.cc:457] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:28.196334 26713 catalog_manager.cc:1776] Deleting table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d] in response to Hive Metastore notification log event 8
I0619 00:12:28.196491 27637 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.624s (starting election)
I0619 00:12:28.196702 27637 raft_consensus.cc:2784] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.196902 27783 ts_tablet_manager.cc:1369] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:28.201763 27761 raft_consensus.cc:325] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.202358 27761 raft_consensus.cc:351] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:28.202538 27761 raft_consensus.cc:628] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Initialized, Role: FOLLOWER
W0619 00:12:28.194217 26876 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.119.1:42295, user_credentials={real_user=jenkins}} blocked reactor thread for 50971.4us
I0619 00:12:28.202989 27761 consensus_queue.cc:227] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.206674 27761 ts_tablet_manager.cc:1073] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I0619 00:12:28.206997 26812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ca7268d23f314af5a87b71a79246c324" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:28.207463 26812 raft_consensus.cc:2784] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.207725 26812 raft_consensus.cc:2732] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.938s (vote granted)
I0619 00:12:28.207775 27784 tablet_replica.cc:261] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:28.208004 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.208191 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.208405 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.208621 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.209040 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.209586 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.210155 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(e63eb8276d6d4a03b30d54126e965dbd)
I0619 00:12:28.210638 27784 raft_consensus.cc:2042] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:28.211230 27784 raft_consensus.cc:2069] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:28.212680 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "954b5c3ea0d44539b2466a5fba4df731" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710" is_pre_election: true
I0619 00:12:28.213138 26949 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.516s (vote granted)
I0619 00:12:28.213376 26949 raft_consensus.cc:2245] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 0.
I0619 00:12:28.201133 27637 raft_consensus.cc:479] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.214282 27637 leader_election.cc:243] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:28.214547 27013 leader_election.cc:379] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:28.215324 27637 leader_election.cc:243] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.216675 26949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "954b5c3ea0d44539b2466a5fba4df731" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f719e8be9c9447a9acd40728c0a5710"
I0619 00:12:28.217069 26949 raft_consensus.cc:2784] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.216965 26810 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "954b5c3ea0d44539b2466a5fba4df731" candidate_uuid: "a2c1dfdaa6d442539e4342d184d2167c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:28.217298 26949 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.737s (vote granted)
I0619 00:12:28.194334 27785 consensus_peers.cc:456] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 -> Peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): Closing peer: a2c1dfdaa6d442539e4342d184d2167c
I0619 00:12:28.223835 27785 consensus_peers.cc:456] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:28.225405 27784 ts_tablet_manager.cc:1369] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:28.226928 26949 raft_consensus.cc:2245] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:28.231374 26810 raft_consensus.cc:2784] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.232097 26810 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.073s (vote granted)
I0619 00:12:28.234872 26535 catalog_manager.cc:3786] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
I0619 00:12:28.234477 26812 raft_consensus.cc:2245] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:28.237206 27785 raft_consensus.cc:2069] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:28.241299 26810 raft_consensus.cc:2245] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c1dfdaa6d442539e4342d184d2167c in term 1.
I0619 00:12:28.242612 27015 leader_election.cc:379] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.242882 27015 leader_election.cc:270] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:28.248203 27791 raft_consensus.cc:2732] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.625s (election complete)
I0619 00:12:28.248507 27791 raft_consensus.cc:2539] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:28.251351 26713 catalog_manager.cc:4025] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: Sending DeleteTablet for 3 replicas of tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7
I0619 00:12:28.251956 27013 leader_election.cc:379] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [CANDIDATE]: Term 1 election: Vote granted by peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248)
I0619 00:12:28.252146 27785 ts_tablet_manager.cc:1369] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:28.252240 26876 leader_election.cc:379] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.252605 26713 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 on 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295) (Table deleted at 2018-06-19 00:12:28 PDT)
I0619 00:12:28.256106 27783 ts_tablet_manager.cc:1382] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:28.258595 26792 tablet_service.cc:800] Processing DeleteTablet for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:28 PDT) from {username='jenkins'} at 127.24.119.1:39548
I0619 00:12:28.262375 27783 log.cc:986] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/e63eb8276d6d4a03b30d54126e965dbd
I0619 00:12:28.261962 27791 raft_consensus.cc:593] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Becoming Leader. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Running, Role: LEADER
I0619 00:12:28.262229 26713 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 on 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248) (Table deleted at 2018-06-19 00:12:28 PDT)
I0619 00:12:28.262991 27791 consensus_queue.cc:204] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.263640 26928 tablet_service.cc:800] Processing DeleteTablet for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:28 PDT) from {username='jenkins'} at 127.24.119.2:46317
I0619 00:12:28.263896 26713 catalog_manager.cc:3113] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 on a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806) (Table deleted at 2018-06-19 00:12:28 PDT)
I0619 00:12:28.264828 27061 tablet_service.cc:800] Processing DeleteTablet for tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-19 00:12:28 PDT) from {username='jenkins'} at 127.24.119.3:56160
I0619 00:12:28.271800 27783 ts_tablet_manager.cc:1403] T e63eb8276d6d4a03b30d54126e965dbd P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:28.273779 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet e63eb8276d6d4a03b30d54126e965dbd (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:28.275579 27784 ts_tablet_manager.cc:1382] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:28.275925 27784 log.cc:986] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/e63eb8276d6d4a03b30d54126e965dbd
I0619 00:12:28.276399 27783 tablet_replica.cc:261] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: stopping tablet replica
I0619 00:12:28.277441 27689 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 1 leader_uuid: "90b46df3f26b4096876d341a82aaf7c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:28.278879 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op CompactRowSetsOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.279532 26534 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.62:33747:
name: "default.table_06c34c9cf69d48a7be4493acace87da2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:28.280068 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MinorDeltaCompactionOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
W0619 00:12:28.280465 26534 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_06c34c9cf69d48a7be4493acace87da2 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:28.280531 27784 ts_tablet_manager.cc:1403] T e63eb8276d6d4a03b30d54126e965dbd P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
I0619 00:12:28.280840 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op MajorDeltaCompactionOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.283092 25052 master-stress-test.cc:470] Killed test thread 1/10
I0619 00:12:28.283257 25052 master-stress-test.cc:468] Killing test thread 2/10
I0619 00:12:28.284829 26539 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.62:33747:
table { table_name: "default.table_4a6e4e28e18547719a48db319fc95750" } new_table_name: "default.table_8a99507c6bce41038f15f6aea10b464e" alter_external_catalogs: true
I0619 00:12:28.285295 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op UndoDeltaBlockGCOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.285529 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushMRSOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.285392 26539 rpcz_store.cc:265] Call kudu.master.MasterService.AlterTable from 127.24.119.62:33747 (request call id 94) took 1743ms. Request Metrics: {}
I0619 00:12:28.285744 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op FlushDeltaMemStoresOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.285923 27783 maintenance_manager.cc:237] P a2c1dfdaa6d442539e4342d184d2167c: Unregistered op LogGCOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.288442 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet e63eb8276d6d4a03b30d54126e965dbd (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:28.289055 27784 tablet_replica.cc:261] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: stopping tablet replica
I0619 00:12:28.290154 27783 raft_consensus.cc:2042] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 LEADER]: Raft consensus shutting down.
I0619 00:12:28.290777 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op CompactRowSetsOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.291018 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MinorDeltaCompactionOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.291245 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op MajorDeltaCompactionOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.290381 27783 consensus_peers.cc:456] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Closing peer: 1f719e8be9c9447a9acd40728c0a5710
I0619 00:12:28.291471 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op UndoDeltaBlockGCOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.291728 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushMRSOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.291736 27783 consensus_peers.cc:456] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Closing peer: 10c4e713d4c44dd8904e5f77ba815785
I0619 00:12:28.291934 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op FlushDeltaMemStoresOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.292130 27784 maintenance_manager.cc:237] P 1f719e8be9c9447a9acd40728c0a5710: Unregistered op LogGCOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.292307 27784 raft_consensus.cc:2042] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:28.292601 27784 raft_consensus.cc:2069] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:28.292579 27783 raft_consensus.cc:2069] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:28.292374 26539 catalog_manager.cc:3786] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3). New cstate: current_term: 1 leader_uuid: "a2c1dfdaa6d442539e4342d184d2167c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: HEALTHY } } }
I0619 00:12:28.294669 27796 tablet.cc:1200] T 912bbbfd578c495ebd11e1eb0d170e7e P 10c4e713d4c44dd8904e5f77ba815785: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:28.310554 27785 ts_tablet_manager.cc:1382] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:28.310982 27785 log.cc:986] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/e63eb8276d6d4a03b30d54126e965dbd
I0619 00:12:28.311954 27783 ts_tablet_manager.cc:1369] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:28.315323 27795 tablet.cc:1200] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:28.294533 27784 ts_tablet_manager.cc:1369] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:28.321135 26534 catalog_manager.cc:4229] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: default.table_f3e7e07d4ed74a6cac218172903cccfd [id=16e5d101e49745c58a820c708268b84b] alter complete (version 1)
I0619 00:12:28.324036 27785 ts_tablet_manager.cc:1403] T e63eb8276d6d4a03b30d54126e965dbd P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:28.325992 27785 tablet_replica.cc:261] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: stopping tablet replica
I0619 00:12:28.326131 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet e63eb8276d6d4a03b30d54126e965dbd (table default.table_d9d85b384d834dd9ac9814f9d0fa9ce3 [id=0bd92425ac744976ad74f66a1de73f9f]) successfully deleted
I0619 00:12:28.314051 27802 tablet.cc:1200] T 912bbbfd578c495ebd11e1eb0d170e7e P a2c1dfdaa6d442539e4342d184d2167c: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:28.333554 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op CompactRowSetsOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.333817 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MinorDeltaCompactionOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.334026 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op MajorDeltaCompactionOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.334237 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op UndoDeltaBlockGCOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.334480 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushMRSOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.334723 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op FlushDeltaMemStoresOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.334930 27785 maintenance_manager.cc:237] P 10c4e713d4c44dd8904e5f77ba815785: Unregistered op LogGCOp(55aa16ffaca64a1ca2dc2e2cb236cfd7)
I0619 00:12:28.335127 27785 raft_consensus.cc:2042] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus shutting down.
I0619 00:12:28.335417 27785 raft_consensus.cc:2069] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Raft consensus is shut down!
I0619 00:12:28.336949 27785 ts_tablet_manager.cc:1369] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Deleting tablet data with delete state TABLET_DATA_DELETED
I0619 00:12:28.339424 27783 ts_tablet_manager.cc:1382] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:28.339730 27783 log.cc:986] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-2/wal/wals/55aa16ffaca64a1ca2dc2e2cb236cfd7
I0619 00:12:28.340298 27783 ts_tablet_manager.cc:1403] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P a2c1dfdaa6d442539e4342d184d2167c: Deleting consensus metadata
I0619 00:12:28.344281 27784 ts_tablet_manager.cc:1382] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:28.346783 27784 log.cc:986] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-1/wal/wals/55aa16ffaca64a1ca2dc2e2cb236cfd7
I0619 00:12:28.347507 27784 ts_tablet_manager.cc:1403] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 1f719e8be9c9447a9acd40728c0a5710: Deleting consensus metadata
W0619 00:12:28.337993 27015 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.1:42295, user_credentials={real_user=jenkins}} blocked reactor thread for 49848.1us
I0619 00:12:28.355389 26521 catalog_manager.cc:3091] TS a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806): tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:28.357837 27796 tablet.cc:1200] T 1764162224124e32922728fe10894601 P 10c4e713d4c44dd8904e5f77ba815785: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
W0619 00:12:28.342068 27013 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.119.2:55248, user_credentials={real_user=jenkins}} blocked reactor thread for 55527.1us
I0619 00:12:28.370797 26534 catalog_manager.cc:4229] T 00000000000000000000000000000000 P 90b46df3f26b4096876d341a82aaf7c7: default.table_8a99507c6bce41038f15f6aea10b464e [id=4accbe5f802d41238340171a9b94219b] alter complete (version 1)
I0619 00:12:28.377053 27771 raft_consensus.cc:457] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0619 00:12:28.377290 27771 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.986s (starting election)
I0619 00:12:28.377538 27771 raft_consensus.cc:479] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.378489 27771 leader_election.cc:243] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.378890 27771 leader_election.cc:243] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.380609 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "65392b375030460abad2b8dcd751a238" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c" is_pre_election: true
I0619 00:12:28.380601 26809 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "65392b375030460abad2b8dcd751a238" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785" is_pre_election: true
I0619 00:12:28.381059 27083 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.903s (vote granted)
I0619 00:12:28.381108 26809 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 1.812s (vote granted)
I0619 00:12:28.381270 27083 raft_consensus.cc:2245] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:28.381332 26809 raft_consensus.cc:2245] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 0.
I0619 00:12:28.382323 26874 leader_election.cc:379] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.382534 26874 leader_election.cc:270] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I0619 00:12:28.382798 26876 leader_election.cc:379] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.382992 27771 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.787s (election complete)
I0619 00:12:28.383244 27771 raft_consensus.cc:2539] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0619 00:12:28.383440 27771 raft_consensus.cc:457] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0619 00:12:28.383633 27771 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.529s (starting election)
I0619 00:12:28.383855 27771 raft_consensus.cc:2784] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.384021 26521 catalog_manager.cc:3091] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:28.389094 27771 raft_consensus.cc:479] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.390013 27771 leader_election.cc:243] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.390444 27771 leader_election.cc:243] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Requesting vote from peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.391516 26809 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "65392b375030460abad2b8dcd751a238" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c4e713d4c44dd8904e5f77ba815785"
I0619 00:12:28.391947 26809 raft_consensus.cc:2784] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.391932 27083 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "65392b375030460abad2b8dcd751a238" candidate_uuid: "1f719e8be9c9447a9acd40728c0a5710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c1dfdaa6d442539e4342d184d2167c"
I0619 00:12:28.392180 26809 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785: Snoozing failure detection for 2.176s (vote granted)
I0619 00:12:28.392341 27083 raft_consensus.cc:2784] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Advancing to term 1
I0619 00:12:28.392563 27083 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c: Snoozing failure detection for 1.641s (vote granted)
I0619 00:12:28.396558 26809 raft_consensus.cc:2245] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:28.396656 27083 raft_consensus.cc:2245] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f719e8be9c9447a9acd40728c0a5710 in term 1.
I0619 00:12:28.398238 26876 leader_election.cc:379] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295)
I0619 00:12:28.398478 26876 leader_election.cc:270] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I0619 00:12:28.398741 26874 leader_election.cc:379] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [CANDIDATE]: Term 1 election: Vote granted by peer a2c1dfdaa6d442539e4342d184d2167c (127.24.119.3:34806)
I0619 00:12:28.399258 27804 tablet.cc:1200] T 1764162224124e32922728fe10894601 P 1f719e8be9c9447a9acd40728c0a5710: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:28.400622 27771 raft_consensus.cc:2732] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710: Snoozing failure detection for 1.545s (election complete)
I0619 00:12:28.400918 27771 raft_consensus.cc:2539] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Leader election won for term 1
I0619 00:12:28.401741 27771 raft_consensus.cc:593] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 LEADER]: Becoming Leader. State: Replica: 1f719e8be9c9447a9acd40728c0a5710, State: Running, Role: LEADER
I0619 00:12:28.402493 27785 ts_tablet_manager.cc:1382] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I0619 00:12:28.402178 27771 consensus_queue.cc:204] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } }
I0619 00:12:28.402845 27785 log.cc:986] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/master-stress-test.0.HmsConfigurations_MasterStressTest.Test_1.1529392303277196-25052/minicluster-data/ts-0/wal/wals/55aa16ffaca64a1ca2dc2e2cb236cfd7
I0619 00:12:28.403422 27785 ts_tablet_manager.cc:1403] T 55aa16ffaca64a1ca2dc2e2cb236cfd7 P 10c4e713d4c44dd8904e5f77ba815785: Deleting consensus metadata
I0619 00:12:28.405251 26523 catalog_manager.cc:3091] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): tablet 55aa16ffaca64a1ca2dc2e2cb236cfd7 (table default.table_f9c29cfbe84f43ff836467ab27c1a7ca [id=edde321eb4574c0a9e3faf347cc90f6d]) successfully deleted
I0619 00:12:28.408929 27805 tablet.cc:1200] T 912bbbfd578c495ebd11e1eb0d170e7e P 1f719e8be9c9447a9acd40728c0a5710: Alter schema from Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 0 to Schema [
	primary key (key),
	10:key[int32 NOT NULL]
] version 1
I0619 00:12:28.416483 26534 catalog_manager.cc:3786] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2). New cstate: current_term: 1 leader_uuid: "1f719e8be9c9447a9acd40728c0a5710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } health_report { overall_health: UNKNOWN } } }
==================
WARNING: ThreadSanitizer: data race (pid=26513)
  Read of size 8 at 0x7ffb5e5b88b8 by thread T65:
    #0 kudu::Status::operator=(kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/status.h:469:7 (libmaster.so+0x10bd00)
    #1 kudu::Synchronizer::StatusCB(kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:44:8 (libmaster.so+0x10bc40)
    #2 kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>::Run(kudu::Synchronizer*, kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:192:12 (libmaster.so+0x10c708)
    #3 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>, void ()(kudu::Synchronizer*, kudu::Status const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>, kudu::Synchronizer*, kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:889:14 (libmaster.so+0x10c5e8)
    #4 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>, void ()(kudu::Synchronizer*, kudu::Status const&), void ()(kudu::internal::UnretainedWrapper<kudu::Synchronizer>)>, void ()(kudu::Synchronizer*, kudu::Status const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1118:12 (libmaster.so+0x10c51a)
    #5 kudu::Callback<void ()(kudu::Status const&)>::Run(kudu::Status const&) const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:436:12 (libmaster.so+0x10b831)
    #6 kudu::master::HmsNotificationLogListenerTask::RunLoop() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/hms_notification_log_listener.cc:136:10 (libmaster.so+0x108e0a)
    #7 boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>::operator()(kudu::master::HmsNotificationLogListenerTask*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libmaster.so+0x110ea9)
    #8 void boost::_bi::list1<boost::_bi::value<kudu::master::HmsNotificationLogListenerTask*> >::operator()<boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libmaster.so+0x110dfa)
    #9 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>, boost::_bi::list1<boost::_bi::value<kudu::master::HmsNotificationLogListenerTask*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libmaster.so+0x110d83)
    #10 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>, boost::_bi::list1<boost::_bi::value<kudu::master::HmsNotificationLogListenerTask*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libmaster.so+0x110b79)
    #11 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1)
    #12 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bd8b4)

  Previous write of size 8 at 0x7ffb5e5b88b8 by thread T24 (mutexes: read M1468):
    #0 boost::intrusive::circular_list_algorithms<boost::intrusive::list_node_traits<void*> >::init(boost::intrusive::list_node<void*>* const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/intrusive/circular_list_algorithms.hpp:72:22 (libkrpc.so+0x99c92)
    #1 boost::intrusive::generic_hook<boost::intrusive::circular_list_algorithms<boost::intrusive::list_node_traits<void*> >, boost::intrusive::dft_tag, (boost::intrusive::link_mode_type)1, (boost::intrusive::base_hook_type)1>::generic_hook() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/intrusive/detail/generic_hook.hpp:174:10 (libkrpc.so+0xc4669)
    #2 boost::intrusive::list_base_hook<void, void, void>::list_base_hook() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/intrusive/list_hook.hpp:83:7 (libkrpc.so+0xc2049)
    #3 kudu::rpc::ReactorTask::ReactorTask() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:262:14 (libkrpc.so+0xbd5fb)
    #4 kudu::rpc::QueueTransferTask::QueueTransferTask(gscoped_ptr<kudu::rpc::OutboundTransfer, kudu::DefaultDeleter<kudu::rpc::OutboundTransfer> >, kudu::rpc::Connection*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/connection.cc:432:3 (libkrpc.so+0x98ea4)
    #5 kudu::rpc::Connection::QueueResponseForCall(gscoped_ptr<kudu::rpc::InboundCall, kudu::DefaultDeleter<kudu::rpc::InboundCall> >) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/connection.cc:474:33 (libkrpc.so+0x94d79)
    #6 kudu::rpc::InboundCall::Respond(google::protobuf::MessageLite const&, bool) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/inbound_call.cc:165:10 (libkrpc.so+0xa11b9)
    #7 kudu::rpc::InboundCall::RespondSuccess(google::protobuf::MessageLite const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/inbound_call.cc:109:3 (libkrpc.so+0xa0fd5)
    #8 kudu::rpc::RpcContext::RespondSuccess() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/rpc_context.cc:77:12 (libkrpc.so+0xd5b35)
    #9 kudu::master::MasterServiceImpl::AlterTable(kudu::master::AlterTableRequestPB const*, kudu::master::AlterTableResponsePB*, kudu::rpc::RpcContext*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_service.cc:350:8 (libmaster.so+0x128903)
    #10 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_13::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/src/kudu/master/master.service.cc:259:13 (libmaster_proto.so+0xe2544)
    #11 _ZNSt3__18__invokeIRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS1_12MetricEntityEERKS4_INS1_3rpc13ResultTrackerEEE4$_13JPKN6google8protobuf7MessageEPSI_PNS9_10RpcContextEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSO_DpOSP_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (libmaster_proto.so+0xe24d1)
    #12 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master15MasterServiceIfC1ERK13scoped_refptrINS3_12MetricEntityEERKS6_INS3_3rpc13ResultTrackerEEE4$_13PKN6google8protobuf7MessageEPSK_PNSB_10RpcContextEEEEvDpOT_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:349 (libmaster_proto.so+0xe24d1)
    #13 std::__1::__function::__func<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_13, std::__1::allocator<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_13>, void ()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libmaster_proto.so+0xe23e4)
    #14 std::__1::function<void ()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xefce9)
    #15 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/service_if.cc:145:3 (libkrpc.so+0xef632)
    #16 kudu::rpc::ServicePool::RunThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/service_pool.cc:225:15 (libkrpc.so+0xf0b8c)
    #17 boost::_mfi::mf0<void, kudu::rpc::ServicePool>::operator()(kudu::rpc::ServicePool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xf2a69)
    #18 void boost::_bi::list1<boost::_bi::value<kudu::rpc::ServicePool*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ServicePool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::rpc::ServicePool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xf29ba)
    #19 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ServicePool>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ServicePool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xf2943)
    #20 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ServicePool>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ServicePool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xf2739)
    #21 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1)
    #22 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bd8b4)

  Location is stack of thread T24.

  Mutex M1468 (0x7b4c00000f40) created at:
    #0 pthread_rwlock_init /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1304 (kudu-master+0x459364)
    #1 kudu::RWMutex::Init(kudu::RWMutex::Priority) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/rw_mutex.cc:78:8 (libkudu_util.so+0x1aa898)
    #2 kudu::RWMutex::RWMutex(kudu::RWMutex::Priority) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/rw_mutex.cc:56:3 (libkudu_util.so+0x1aaad3)
    #3 kudu::master::CatalogManager::CatalogManager(kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:667:5 (libmaster.so+0xac687)
    #4 kudu::master::Master::Master(kudu::master::MasterOptions const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:122:26 (libmaster.so+0x111cbf)
    #5 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:79:10 (kudu-master+0x4cb48e)
    #6 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb16e)

  Thread T65 'hms-notificatio' (tid=26713, running) created by thread T59 at:
    #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (kudu-master+0x45aebb)
    #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1bd2df)
    #2 kudu::Status kudu::Thread::Create<void (kudu::master::HmsNotificationLogListenerTask::*)(), kudu::master::HmsNotificationLogListenerTask*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::master::HmsNotificationLogListenerTask::* const&)(), kudu::master::HmsNotificationLogListenerTask* const&, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libmaster.so+0x10b515)
    #3 kudu::master::HmsNotificationLogListenerTask::Init() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/hms_notification_log_listener.cc:94:10 (libmaster.so+0x108c33)
    #4 kudu::master::CatalogManager::Init(bool) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:714:5 (libmaster.so+0xad0df)
    #5 kudu::master::Master::InitCatalogManager() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:216:3 (libmaster.so+0x11332f)
    #6 kudu::master::Master::InitCatalogManagerTask() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:205:14 (libmaster.so+0x113182)
    #7 kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>::Run(kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x117e89)
    #8 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, void ()(kudu::master::Master*)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:873:14 (libmaster.so+0x117da5)
    #9 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, void ()(kudu::master::Master*), void ()(kudu::internal::UnretainedWrapper<kudu::master::Master>)>, void ()(kudu::master::Master*)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x117cea)
    #10 kudu::Callback<void ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa2d2d)
    #11 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1ca42d)
    #12 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c6158)
    #13 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d10c9)
    #14 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d101a)
    #15 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d0fa3)
    #16 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d0d99)
    #17 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1)
    #18 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bd8b4)

  Thread T24 'rpc worker-2653' (tid=26539, running) created by main thread at:
    #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (kudu-master+0x45aebb)
    #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1bd2df)
    #2 kudu::Status kudu::Thread::Create<void (kudu::rpc::ServicePool::*)(), kudu::rpc::ServicePool*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkrpc.so+0xf1e55)
    #3 kudu::rpc::ServicePool::Init(int) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0xf0796)
    #4 kudu::RpcServer::RegisterService(gscoped_ptr<kudu::rpc::ServiceIf, kudu::DefaultDeleter<kudu::rpc::ServiceIf> >) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/server/rpc_server.cc:157:3 (libserver_process.so+0x76c39)
    #5 kudu::server::ServerBase::RegisterService(gscoped_ptr<kudu::rpc::ServiceIf, kudu::DefaultDeleter<kudu::rpc::ServiceIf> >) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/server/server_base.cc:612:23 (libserver_process.so+0x7de7f)
    #6 kudu::master::Master::StartAsync() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:187:3 (libmaster.so+0x1129ef)
    #7 kudu::master::Master::Start() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:170:3 (libmaster.so+0x1127f5)
    #8 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:84:3 (kudu-master+0x4cb534)
    #9 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb16e)

SUMMARY: ThreadSanitizer: data race /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/status.h:469:7 in kudu::Status::operator=(kudu::Status const&)
==================
FATAL: ThreadSanitizer CHECK failed: /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_allocator_secondary.h:242 "((IsAligned(p, page_size_))) != (0)" (0x0, 0x0)
    #0 __tsan::TsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_rtl_report.cc:48 (kudu-master+0x49d703)
    #1 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:79 (kudu-master+0x4bae75)
    #2 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::ReturnNullOrDieOnFailure>::GetHeader(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_allocator_secondary.h:242 (kudu-master+0x48c5fc)
    #3 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::ReturnNullOrDieOnFailure>::GetHeader(void const*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_allocator_secondary.h:246 (kudu-master+0x48c5fc)
    #4 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::ReturnNullOrDieOnFailure>::Deallocate(__sanitizer::AllocatorStats*, void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_allocator_secondary.h:81 (kudu-master+0x48c5fc)
    #5 __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__tsan::AP64>, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<__tsan::AP64> >, __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::ReturnNullOrDieOnFailure> >::Deallocate(__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<__tsan::AP64> >*, void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_allocator_combined.h:89 (kudu-master+0x48c5fc)
    #6 __tsan::user_free(__tsan::ThreadState*, unsigned long, void*, bool) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_mman.cc:171 (kudu-master+0x48c5fc)
    #7 operator delete[](void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:125 (kudu-master+0x4c8a04)
    #8 kudu::Status::operator=(kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/status.h:470:5 (libmaster.so+0x10bd20)
    #9 kudu::Synchronizer::StatusCB(kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:44:8 (libmaster.so+0x10bc40)
    #10 kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>::Run(kudu::Synchronizer*, kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:192:12 (libmaster.so+0x10c708)
    #11 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>, void ()(kudu::Synchronizer*, kudu::Status const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>, kudu::Synchronizer*, kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:889:14 (libmaster.so+0x10c5e8)
    #12 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::Synchronizer::*)(kudu::Status const&)>, void ()(kudu::Synchronizer*, kudu::Status const&), void ()(kudu::internal::UnretainedWrapper<kudu::Synchronizer>)>, void ()(kudu::Synchronizer*, kudu::Status const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1118:12 (libmaster.so+0x10c51a)
    #13 kudu::Callback<void ()(kudu::Status const&)>::Run(kudu::Status const&) const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:436:12 (libmaster.so+0x10b831)
    #14 kudu::master::HmsNotificationLogListenerTask::RunLoop() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/hms_notification_log_listener.cc:136:10 (libmaster.so+0x108e0a)
    #15 boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>::operator()(kudu::master::HmsNotificationLogListenerTask*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libmaster.so+0x110ea9)
    #16 void boost::_bi::list1<boost::_bi::value<kudu::master::HmsNotificationLogListenerTask*> >::operator()<boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libmaster.so+0x110dfa)
    #17 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>, boost::_bi::list1<boost::_bi::value<kudu::master::HmsNotificationLogListenerTask*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libmaster.so+0x110d83)
    #18 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::master::HmsNotificationLogListenerTask>, boost::_bi::list1<boost::_bi::value<kudu::master::HmsNotificationLogListenerTask*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libmaster.so+0x110b79)
    #19 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1)
    #20 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bd8b4)
    #21 __tsan_thread_start_func /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:982 (kudu-master+0x4298bd)
    #22 start_thread <null> (libpthread.so.0+0x3ef6a079d0)
    #23 clone <null> (libc.so.6+0x3ef66e88fc)

W0619 00:12:28.504262 26734 connection.cc:512] client connection to 127.24.119.62:58195 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.62:58195): Connection reset by peer (error 104)
W0619 00:12:28.505172 26875 connection.cc:512] client connection to 127.24.119.62:58195 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.62:58195): Connection reset by peer (error 104)
W0619 00:12:28.507397 27014 connection.cc:512] client connection to 127.24.119.62:58195 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.62:58195): Connection reset by peer (error 104)
W0619 00:12:28.507956 27153 client-internal.cc:218] Unable to send the request (table { table_name: "default.table_f9c29cfbe84f43ff836467ab27c1a7ca" }) to leader Master (127.24.119.62:58195): Network error: recv got EOF from 127.24.119.62:58195 (error 108)
I0619 00:12:28.508322 27153 client-internal.cc:223] Determining the new leader Master and retrying...
W0619 00:12:28.508314 27148 client-internal.cc:218] Unable to send the request (name: "default.table_3a31a472b2e1460591b55ec649f29582" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } }) to leader Master (127.24.119.62:58195): Network error: recv got EOF from 127.24.119.62:58195 (error 108)
I0619 00:12:28.509142 27148 client-internal.cc:223] Determining the new leader Master and retrying...
W0619 00:12:28.510260 27152 client-internal.cc:218] Unable to send the request (table { table_name: "default.table_31d388c40f064a2a97bbbec12c77fbd9" } new_table_name: "default.table_f3e7e07d4ed74a6cac218172903cccfd" alter_external_catalogs: true) to leader Master (127.24.119.62:58195): Network error: recv got EOF from 127.24.119.62:58195 (error 108)
I0619 00:12:28.510665 27152 client-internal.cc:223] Determining the new leader Master and retrying...
I0619 00:12:28.511190 27149 client-internal.cc:223] Determining the new leader Master and retrying...
I0619 00:12:28.513684 27154 client-internal.cc:223] Determining the new leader Master and retrying...
I0619 00:12:28.514699 27150 client-internal.cc:223] Determining the new leader Master and retrying...
I0619 00:12:28.673527 26809 raft_consensus.cc:1081] T ca7268d23f314af5a87b71a79246c324 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:28.674584 27771 consensus_queue.cc:963] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:28.685088 26855 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.685350 27083 raft_consensus.cc:1081] T ca7268d23f314af5a87b71a79246c324 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:28.687559 27782 consensus_queue.cc:963] T ca7268d23f314af5a87b71a79246c324 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:28.691726 26855 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.693619 26995 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.705153 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.706303 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.721304 26809 raft_consensus.cc:1081] T 954b5c3ea0d44539b2466a5fba4df731 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:28.722987 27800 consensus_queue.cc:963] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:28.727455 26855 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.735046 26948 raft_consensus.cc:1081] T 954b5c3ea0d44539b2466a5fba4df731 P 1f719e8be9c9447a9acd40728c0a5710 [term 1 FOLLOWER]: Refusing update from remote peer a2c1dfdaa6d442539e4342d184d2167c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:28.736793 27791 consensus_queue.cc:963] T 954b5c3ea0d44539b2466a5fba4df731 P a2c1dfdaa6d442539e4342d184d2167c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:28.740449 26855 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.742317 26995 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.743283 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.747468 26995 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.861238 27083 raft_consensus.cc:1081] T 65392b375030460abad2b8dcd751a238 P a2c1dfdaa6d442539e4342d184d2167c [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0619 00:12:28.862182 27782 consensus_queue.cc:963] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:28.865146 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.873100 26995 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.873204 26809 raft_consensus.cc:1081] T 65392b375030460abad2b8dcd751a238 P 10c4e713d4c44dd8904e5f77ba815785 [term 1 FOLLOWER]: Refusing update from remote peer 1f719e8be9c9447a9acd40728c0a5710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
W0619 00:12:28.875504 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.875530 27782 consensus_queue.cc:963] T 65392b375030460abad2b8dcd751a238 P 1f719e8be9c9447a9acd40728c0a5710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0619 00:12:28.879117 26855 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.881115 26855 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.929154 27819 raft_consensus.cc:457] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:28.929219 27820 raft_consensus.cc:457] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:28.929450 27819 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.568s (starting election)
I0619 00:12:28.929625 27820 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 0.544s (starting election)
I0619 00:12:28.929695 27819 raft_consensus.cc:479] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:28.929899 27820 raft_consensus.cc:479] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:28.930434 27819 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:28.930742 27819 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:28.930886 27820 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:28.931852 27582 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 194 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696" is_pre_election: true
I0619 00:12:28.932329 27582 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 1.521s (vote granted)
I0619 00:12:28.932538 27582 raft_consensus.cc:2245] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8582cfc277d4d4597c5605b7ac0ed48 in term 1.
I0619 00:12:28.933461 27254 leader_election.cc:379] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:28.933712 27254 leader_election.cc:270] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won.
I0619 00:12:28.934244 27819 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 1.058s (election complete)
I0619 00:12:28.934476 27819 raft_consensus.cc:2539] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I0619 00:12:28.934711 27819 raft_consensus.cc:457] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 90b46df3f26b4096876d341a82aaf7c7)
I0619 00:12:28.934937 27819 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 1.140s (starting election)
I0619 00:12:28.935154 27819 raft_consensus.cc:2784] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 1 FOLLOWER]: Advancing to term 2
I0619 00:12:28.936638 27820 leader_election.cc:243] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000)
I0619 00:12:28.937718 27342 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "555bff44d4f942eaad13f790cca0a696" candidate_term: 2 candidate_status { last_received { term: 1 index: 194 } } ignore_live_leader: false dest_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" is_pre_election: true
W0619 00:12:28.939101 27253 leader_election.cc:299] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195): Network error: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:28.937515 27525 leader_election.cc:299] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195): Network error: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.941908 27819 raft_consensus.cc:479] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:28.942869 27342 raft_consensus.cc:2175] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 in current term 2: Already voted for candidate b8582cfc277d4d4597c5605b7ac0ed48 in this term.
I0619 00:12:28.942562 27819 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 election: Requesting vote from peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195)
I0619 00:12:28.945050 27819 leader_election.cc:243] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 election: Requesting vote from peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:28.945194 27545 leader_election.cc:393] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b8582cfc277d4d4597c5605b7ac0ed48 (127.24.119.61:43000). Message: Invalid argument: T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 555bff44d4f942eaad13f790cca0a696 in current term 2: Already voted for candidate b8582cfc277d4d4597c5605b7ac0ed48 in this term.
I0619 00:12:28.945444 27545 leader_election.cc:270] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
W0619 00:12:28.945996 27253 leader_election.cc:299] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195): Network error: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:28.946331 27820 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 1.088s (election complete)
I0619 00:12:28.946377 27582 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 194 } } ignore_live_leader: false dest_uuid: "555bff44d4f942eaad13f790cca0a696"
I0619 00:12:28.946597 27820 raft_consensus.cc:2784] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 1 FOLLOWER]: Advancing to term 2
I0619 00:12:28.951916 27820 raft_consensus.cc:2487] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0619 00:12:28.952219 27582 raft_consensus.cc:2732] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696: Snoozing failure detection for 2.290s (vote granted)
I0619 00:12:28.954715 27582 raft_consensus.cc:2245] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8582cfc277d4d4597c5605b7ac0ed48 in term 2.
I0619 00:12:28.955432 27254 leader_election.cc:379] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 election: Vote granted by peer 555bff44d4f942eaad13f790cca0a696 (127.24.119.60:33918)
I0619 00:12:28.955601 27254 leader_election.cc:270] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
I0619 00:12:28.955930 27819 raft_consensus.cc:2732] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Snoozing failure detection for 0.784s (election complete)
I0619 00:12:28.956202 27819 raft_consensus.cc:2539] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 2 FOLLOWER]: Leader election won for term 2
I0619 00:12:28.956933 27819 raft_consensus.cc:593] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [term 2 LEADER]: Becoming Leader. State: Replica: b8582cfc277d4d4597c5605b7ac0ed48, State: Running, Role: LEADER
I0619 00:12:28.957442 27819 consensus_queue.cc:204] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 1.194, Last appended by leader: 194, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } }
I0619 00:12:28.959280 27823 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8582cfc277d4d4597c5605b7ac0ed48. Latest consensus state: current_term: 2 leader_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:28.959717 27823 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:28.961071 27824 catalog_manager.cc:1006] Loading table and tablet metadata into memory...
I0619 00:12:28.968268 27824 catalog_manager.cc:323] Loaded metadata for table default.table_f3e7e07d4ed74a6cac218172903cccfd [id=16e5d101e49745c58a820c708268b84b]
I0619 00:12:28.968689 27824 catalog_manager.cc:323] Loaded metadata for table default.table_8a99507c6bce41038f15f6aea10b464e [id=4accbe5f802d41238340171a9b94219b]
I0619 00:12:28.969061 27824 catalog_manager.cc:323] Loaded metadata for table default.table_3a31a472b2e1460591b55ec649f29582 [id=5c424f62ca27467eb104ac30e5e4ae56]
I0619 00:12:28.969442 27824 catalog_manager.cc:323] Loaded metadata for table default.table_f4f93d7c11cf4735980b09abd2aa7f62 [id=602806a614894f7c8b6d41137a33dcd5]
I0619 00:12:28.969866 27824 catalog_manager.cc:323] Loaded metadata for table default.table_06c34c9cf69d48a7be4493acace87da2 [id=9354ca9558ad49bb92a441c542c973cd]
I0619 00:12:28.979600 27824 catalog_manager.cc:377] Loaded metadata for tablet 1764162224124e32922728fe10894601 (table default.table_8a99507c6bce41038f15f6aea10b464e [id=4accbe5f802d41238340171a9b94219b])
I0619 00:12:28.985433 27824 catalog_manager.cc:377] Loaded metadata for tablet 65392b375030460abad2b8dcd751a238 (table default.table_06c34c9cf69d48a7be4493acace87da2 [id=9354ca9558ad49bb92a441c542c973cd])
I0619 00:12:28.989038 27824 catalog_manager.cc:377] Loaded metadata for tablet 912bbbfd578c495ebd11e1eb0d170e7e (table default.table_f3e7e07d4ed74a6cac218172903cccfd [id=16e5d101e49745c58a820c708268b84b])
I0619 00:12:28.990206 27824 catalog_manager.cc:377] Loaded metadata for tablet 954b5c3ea0d44539b2466a5fba4df731 (table default.table_f4f93d7c11cf4735980b09abd2aa7f62 [id=602806a614894f7c8b6d41137a33dcd5])
I0619 00:12:28.996217 27824 catalog_manager.cc:377] Loaded metadata for tablet ca7268d23f314af5a87b71a79246c324 (table default.table_3a31a472b2e1460591b55ec649f29582 [id=5c424f62ca27467eb104ac30e5e4ae56])
I0619 00:12:29.001207 27824 catalog_manager.cc:1017] Initializing Kudu internal certificate authority...
I0619 00:12:29.007247 27824 catalog_manager.cc:1026] Loading token signing keys...
I0619 00:12:29.012367 27824 catalog_manager.cc:4101] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48: Loaded TSK: 0
I0619 00:12:29.017417 27582 raft_consensus.cc:1081] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [term 2 FOLLOWER]: Refusing update from remote peer b8582cfc277d4d4597c5605b7ac0ed48: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 2 index: 196. (index mismatch)
W0619 00:12:29.017729 27253 consensus_peers.cc:434] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 -> Peer 90b46df3f26b4096876d341a82aaf7c7 (127.24.119.62:58195): Couldn't send request to peer 90b46df3f26b4096876d341a82aaf7c7 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times.
I0619 00:12:29.018347 27819 consensus_queue.cc:963] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I0619 00:12:29.021569 27820 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8582cfc277d4d4597c5605b7ac0ed48. Latest consensus state: current_term: 2 leader_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:29.022090 27820 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:29.024679 27819 sys_catalog.cc:337] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:29.025354 27824 catalog_manager.cc:1037] Loading latest processed Hive Metastore notification log event ID...
I0619 00:12:29.025595 27819 sys_catalog.cc:340] T 00000000000000000000000000000000 P b8582cfc277d4d4597c5605b7ac0ed48 [sys.catalog]: This master's current role is: LEADER
I0619 00:12:29.025636 27820 sys_catalog.cc:337] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:29.026240 27820 sys_catalog.cc:340] T 00000000000000000000000000000000 P 555bff44d4f942eaad13f790cca0a696 [sys.catalog]: This master's current role is: FOLLOWER
I0619 00:12:29.085489 27323 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.61:49954:
table { table_name: "default.table_f9c29cfbe84f43ff836467ab27c1a7ca" }
I0619 00:12:29.089828 27322 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.24.119.61:49954:
table { table_name: "default.table_d9d85b384d834dd9ac9814f9d0fa9ce3" }
I0619 00:12:29.184978 27321 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.61:49954:
name: "default.table_f4f93d7c11cf4735980b09abd2aa7f62"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:29.187243 27321 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f4f93d7c11cf4735980b09abd2aa7f62 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:29.187829 27318 catalog_manager.cc:2136] Servicing AlterTable request from {username='jenkins'} at 127.24.119.61:49954:
table { table_name: "default.table_31d388c40f064a2a97bbbec12c77fbd9" } new_table_name: "default.table_f3e7e07d4ed74a6cac218172903cccfd" alter_external_catalogs: true
I0619 00:12:29.190217 27320 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.61:49954:
name: "default.table_c27e1e2853484443811ecccd92146f54"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0619 00:12:29.190414 27319 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.24.119.61:49954:
name: "default.table_3a31a472b2e1460591b55ec649f29582"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0619 00:12:29.191790 27320 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c27e1e2853484443811ecccd92146f54 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
W0619 00:12:29.191987 27319 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3a31a472b2e1460591b55ec649f29582 in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended).
I0619 00:12:29.193440 25052 master-stress-test.cc:470] Killed test thread 2/10
I0619 00:12:29.193645 25052 master-stress-test.cc:468] Killing test thread 3/10
I0619 00:12:29.194053 25052 master-stress-test.cc:470] Killed test thread 3/10
I0619 00:12:29.194198 25052 master-stress-test.cc:468] Killing test thread 4/10
W0619 00:12:29.195739 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:29.222493 26995 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:29.256420 26996 heartbeater.cc:467] Master 127.24.119.61:43000 was elected leader, sending a full tablet report...
W0619 00:12:29.259366 26995 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:29.277118 25052 master-stress-test.cc:470] Killed test thread 4/10
I0619 00:12:29.277304 25052 master-stress-test.cc:468] Killing test thread 5/10
I0619 00:12:29.277523 25052 master-stress-test.cc:470] Killed test thread 5/10
I0619 00:12:29.277664 25052 master-stress-test.cc:468] Killing test thread 6/10
I0619 00:12:29.277894 25052 master-stress-test.cc:470] Killed test thread 6/10
I0619 00:12:29.278033 25052 master-stress-test.cc:468] Killing test thread 7/10
I0619 00:12:29.278209 25052 master-stress-test.cc:470] Killed test thread 7/10
I0619 00:12:29.278348 25052 master-stress-test.cc:468] Killing test thread 8/10
I0619 00:12:29.278404 27689 hms_notification_log_listener.cc:224] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 555bff44d4f942eaad13f790cca0a696, Raft Consensus state: current_term: 2 leader_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90b46df3f26b4096876d341a82aaf7c7" member_type: VOTER last_known_addr { host: "127.24.119.62" port: 58195 } } peers { permanent_uuid: "b8582cfc277d4d4597c5605b7ac0ed48" member_type: VOTER last_known_addr { host: "127.24.119.61" port: 43000 } } peers { permanent_uuid: "555bff44d4f942eaad13f790cca0a696" member_type: VOTER last_known_addr { host: "127.24.119.60" port: 33918 } } }
I0619 00:12:29.278528 25052 master-stress-test.cc:470] Killed test thread 8/10
I0619 00:12:29.278662 25052 master-stress-test.cc:468] Killing test thread 9/10
I0619 00:12:29.278901 25052 master-stress-test.cc:470] Killed test thread 9/10
I0619 00:12:29.279042 25052 master-stress-test.cc:468] Killing test thread 10/10
I0619 00:12:29.279220 25052 master-stress-test.cc:470] Killed test thread 10/10
I0619 00:12:29.279453 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 26703
I0619 00:12:29.315378 26928 tablet_service.cc:759] Processing CreateTablet for tablet 721ea0b211084b7c8db28dc69919c430 (table=default.table_c27e1e2853484443811ecccd92146f54 [id=41b6e709b7d743f39a1801be6eee2e66]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:29.316002 26928 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 721ea0b211084b7c8db28dc69919c430. 1 dirs total, 0 dirs full, 0 dirs failed
I0619 00:12:29.323206 27061 tablet_service.cc:759] Processing CreateTablet for tablet 721ea0b211084b7c8db28dc69919c430 (table=default.table_c27e1e2853484443811ecccd92146f54 [id=41b6e709b7d743f39a1801be6eee2e66]), partition=RANGE (key) PARTITION UNBOUNDED
I0619 00:12:29.323782 27061 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 721ea0b211084b7c8db28dc69919c430. 1 dirs total, 0 dirs full, 0 dirs failed
W0619 00:12:29.324044 27838 negotiation.cc:313] Failed RPC negotiation. Trace:
0619 00:12:29.296102 (+     0us) reactor.cc:583] Submitting negotiation task for client connection to 127.24.119.1:42295
0619 00:12:29.305683 (+  9581us) negotiation.cc:98] Waiting for socket to connect
0619 00:12:29.305751 (+    68us) client_negotiation.cc:160] Beginning negotiation
0619 00:12:29.305885 (+   134us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request
0619 00:12:29.323266 (+ 17381us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.119.1:42295: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":9288,"thread_start_us":6547,"threads_started":1}
W0619 00:12:29.328851 27254 catalog_manager.cc:2855] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Create Tablet RPC failed for tablet 721ea0b211084b7c8db28dc69919c430: Network error: Client connection negotiation failed: client connection to 127.24.119.1:42295: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0619 00:12:29.329113 27254 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 721ea0b211084b7c8db28dc69919c430 on TS 10c4e713d4c44dd8904e5f77ba815785 with a delay of 21 ms (attempt = 1)
W0619 00:12:29.332072 27015 connection.cc:512] client connection to 127.24.119.1:42295 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:42295): Connection reset by peer (error 104)
W0619 00:12:29.332159 26876 connection.cc:512] client connection to 127.24.119.1:42295 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:42295): Connection reset by peer (error 104)
W0619 00:12:29.332202 26874 connection.cc:512] server connection from 127.24.119.1:42665 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:42665): Connection reset by peer (error 104)
W0619 00:12:29.332886 27014 connection.cc:512] server connection from 127.24.119.1:47953 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:47953): Connection reset by peer (error 104)
W0619 00:12:29.332957 27282 connection.cc:512] server connection from 127.24.119.1:48811 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:48811): Connection reset by peer (error 104)
I0619 00:12:29.324442 26928 ts_tablet_manager.cc:1173] T 721ea0b211084b7c8db28dc69919c430 P 1f719e8be9c9447a9acd40728c0a5710: Registered tablet (data state: TABLET_DATA_READY)
W0619 00:12:29.333890 27525 connection.cc:512] server connection from 127.24.119.1:37713 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.1:37713): Connection reset by peer (error 104)
I0619 00:12:29.333974 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 26859
I0619 00:12:29.335031 27061 ts_tablet_manager.cc:1173] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c: Registered tablet (data state: TABLET_DATA_READY)
I0619 00:12:29.336349 27842 ts_tablet_manager.cc:1028] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrapping tablet
I0619 00:12:29.337031 27842 tablet_bootstrap.cc:427] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c: Bootstrap starting.
I0619 00:12:29.338973 27842 tablet_bootstrap.cc:578] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c: No blocks or log segments found. Creating new log.
I0619 00:12:29.341133 27842 tablet_bootstrap.cc:427] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c: No bootstrap required, opened a new log
I0619 00:12:29.341338 27842 ts_tablet_manager.cc:1045] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I0619 00:12:29.342234 27842 raft_consensus.cc:325] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:29.342645 27842 raft_consensus.cc:351] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0619 00:12:29.342833 27842 raft_consensus.cc:628] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c1dfdaa6d442539e4342d184d2167c, State: Initialized, Role: FOLLOWER
I0619 00:12:29.343166 27842 consensus_queue.cc:227] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f719e8be9c9447a9acd40728c0a5710" member_type: VOTER last_known_addr { host: "127.24.119.2" port: 55248 } } peers { permanent_uuid: "a2c1dfdaa6d442539e4342d184d2167c" member_type: VOTER last_known_addr { host: "127.24.119.3" port: 34806 } } peers { permanent_uuid: "10c4e713d4c44dd8904e5f77ba815785" member_type: VOTER last_known_addr { host: "127.24.119.1" port: 42295 } }
I0619 00:12:29.344041 27130 heartbeater.cc:467] Master 127.24.119.61:43000 was elected leader, sending a full tablet report...
W0619 00:12:29.345305 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
I0619 00:12:29.346741 27842 ts_tablet_manager.cc:1073] T 721ea0b211084b7c8db28dc69919c430 P a2c1dfdaa6d442539e4342d184d2167c: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
W0619 00:12:29.347307 27127 heartbeater.cc:587] Failed to heartbeat to 127.24.119.62:58195: Network error: Failed to ping master at 127.24.119.62:58195: Client connection negotiation failed: client connection to 127.24.119.62:58195: connect: Connection refused (error 111)
W0619 00:12:29.352947 27254 catalog_manager.cc:2855] TS 10c4e713d4c44dd8904e5f77ba815785 (127.24.119.1:42295): Create Tablet RPC failed for tablet 721ea0b211084b7c8db28dc69919c430: Network error: Client connection negotiation failed: client connection to 127.24.119.1:42295: connect: Connection refused (error 111)
I0619 00:12:29.353186 27254 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 721ea0b211084b7c8db28dc69919c430 on TS 10c4e713d4c44dd8904e5f77ba815785 with a delay of 52 ms (attempt = 2)
W0619 00:12:29.370909 27251 catalog_manager.cc:2855] TS 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Create Tablet RPC failed for tablet 721ea0b211084b7c8db28dc69919c430: Network error: recv got EOF from 127.24.119.2:55248 (error 108)
W0619 00:12:29.370924 27013 connection.cc:512] client connection to 127.24.119.2:55248 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
I0619 00:12:29.371160 27251 catalog_manager.cc:2891] Scheduling retry of CreateTablet RPC for tablet 721ea0b211084b7c8db28dc69919c430 on TS 1f719e8be9c9447a9acd40728c0a5710 with a delay of 34 ms (attempt = 1)
W0619 00:12:29.371206 27013 consensus_peers.cc:434] T 1764162224124e32922728fe10894601 P a2c1dfdaa6d442539e4342d184d2167c -> Peer 1f719e8be9c9447a9acd40728c0a5710 (127.24.119.2:55248): Couldn't send request to peer 1f719e8be9c9447a9acd40728c0a5710 for tablet 1764162224124e32922728fe10894601. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times.
W0619 00:12:29.371747 27013 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0619 00:12:29.372211 27015 connection.cc:512] server connection from 127.24.119.2:51666 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.2:51666): Connection reset by peer (error 104)
W0619 00:12:29.372969 27251 connection.cc:512] server connection from 127.24.119.2:59374 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.2:59374): Connection reset by peer (error 104)
W0619 00:12:29.373018 27525 connection.cc:512] server connection from 127.24.119.2:33966 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.2:33966): Connection reset by peer (error 104)
I0619 00:12:29.373737 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 26999
W0619 00:12:29.397684 27251 connection.cc:512] server connection from 127.24.119.3:49367 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.3:49367): Connection reset by peer (error 104)
W0619 00:12:29.398732 27545 connection.cc:512] server connection from 127.24.119.3:35086 recv error: Network error: failed to read from TLS socket (remote: 127.24.119.3:35086): Connection reset by peer (error 104)
I0619 00:12:29.398766 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 27176
I0619 00:12:29.417260 25052 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 27474
Shutting down hive metastore.
I0619 00:12:29.875766 25052 master-stress-test.cc:476] Tables created: 5
I0619 00:12:29.875893 25052 master-stress-test.cc:477] Tables altered: 0
I0619 00:12:29.876013 25052 master-stress-test.cc:478] Tables deleted: 0
I0619 00:12:29.876116 25052 master-stress-test.cc:479] Tablets replaced: 61
I0619 00:12:29.876216 25052 master-stress-test.cc:480] Masters restarted: 2
[       OK ] HmsConfigurations/MasterStressTest.Test/1 (28239 ms)
[----------] 2 tests from HmsConfigurations/MasterStressTest (46552 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (46553 ms total)
[  PASSED  ] 2 tests.