Note: This is test shard 2 of 4.
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from HmsConfigurations/MasterFailoverTest
[ RUN      ] HmsConfigurations/MasterFailoverTest.TestCreateTableSync/1
2019-02-27 17:23:32: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1
log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hive-498021fa15186aee8b282d3c032fbd2cede6bec4/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2811 T1 oahu.NativeCodeLoader.<clinit> WARN Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting hive metastore on port 0
3370 T1 odu.Log4JLogger.info Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
3371 T1 odu.Log4JLogger.info Property datanucleus.cache.level2 unknown - will be ignored
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0227 17:23:49.254667 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:54091
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--hive_metastore_uris=thrift://127.0.0.1:43740 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:23:49.346954 140242 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:23:49.347180 140242 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:23:49.348306 140242 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:43740
W0227 17:23:49.348326 140242 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:23:49.348343 140242 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:23:49.348357 140242 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:23:49.348376 140242 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:23:49.350519 140242 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43740
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:54091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.140242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:23:49.357970 140242 master_main.cc:80] Initializing master server...
I0227 17:23:49.358805 140242 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 165775us
I0227 17:23:49.360255 140242 fs_manager.cc:263] Metadata directory not provided
I0227 17:23:49.360317 140242 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:23:49.360630 140242 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:23:49.360663 140242 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:23:49.362943 140242 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/instance:
uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.363267 140242 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal/instance:
uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.371220 140242 fs_manager.cc:503] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.002s
I0227 17:23:49.371829 140242 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:23:49.372272 140242 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:23:49.374632 140242 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:23:49.374712 140242 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/data to 2721 blocks
I0227 17:23:49.376796 140242 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I0227 17:23:49.376871 140242 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.376958 140242 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:23:49.448021 140242 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:23:49.450112 140242 master_main.cc:83] Starting Master server...
I0227 17:23:49.456899 140242 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.254:54091
I0227 17:23:49.457255 140242 webserver.cc:175] Starting webserver on 127.112.245.254:0
I0227 17:23:49.457271 140242 webserver.cc:186] Document root disabled
I0227 17:23:49.457832 140242 webserver.cc:313] Webserver started. Bound to: http://127.112.245.254:34585/
I0227 17:23:49.458364 140242 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
I0227 17:23:49.463563 140492 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:23:49.466567 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 140242
I0227 17:23:49.467187 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:50162
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--hive_metastore_uris=thrift://127.0.0.1:43740 with env {}
I0227 17:23:49.470758 140492 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:23:49.488718 140492 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:23:49.490006 140492 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:50162: Network error: Client connection negotiation failed: client connection to 127.112.245.253:50162: connect: Connection refused (error 111)
I0227 17:23:49.509310 140492 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } attempt: 1
W0227 17:23:49.510226 140492 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:50162: Network error: Client connection negotiation failed: client connection to 127.112.245.253:50162: connect: Connection refused (error 111)
I0227 17:23:49.558374 140492 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } attempt: 2
W0227 17:23:49.559214 140492 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:50162: Network error: Client connection negotiation failed: client connection to 127.112.245.253:50162: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:23:49.574581 140495 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:23:49.574875 140495 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:23:49.576246 140495 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:43740
W0227 17:23:49.576272 140495 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:23:49.576339 140495 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:23:49.576360 140495 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:23:49.576377 140495 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:23:49.579180 140495 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43740
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:50162
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.140495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:23:49.590281 140495 master_main.cc:80] Initializing master server...
I0227 17:23:49.591261 140495 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 165775us
I0227 17:23:49.593163 140495 fs_manager.cc:263] Metadata directory not provided
I0227 17:23:49.593261 140495 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:23:49.593647 140495 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:23:49.593714 140495 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:23:49.597115 140495 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/instance:
uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.597604 140495 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal/instance:
uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.607465 140495 fs_manager.cc:503] Time spent creating directory manager: real 0.009s	user 0.004s	sys 0.004s
I0227 17:23:49.608412 140495 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:23:49.609097 140495 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:23:49.611565 140495 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:23:49.611658 140495 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/data to 2721 blocks
I0227 17:23:49.613342 140495 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I0227 17:23:49.613462 140495 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.613564 140495 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:23:49.638754 140495 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:23:49.639645 140495 master_main.cc:83] Starting Master server...
I0227 17:23:49.647336 140495 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.253:50162
I0227 17:23:49.647882 140495 webserver.cc:175] Starting webserver on 127.112.245.253:0
I0227 17:23:49.647934 140495 webserver.cc:186] Document root disabled
I0227 17:23:49.648703 140495 webserver.cc:313] Webserver started. Bound to: http://127.112.245.253:51624/
I0227 17:23:49.649487 140495 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
I0227 17:23:49.656333 140624 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:23:49.658483 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 140495
I0227 17:23:49.659689 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:43350
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--hive_metastore_uris=thrift://127.0.0.1:43740 with env {}
I0227 17:23:49.665355 140624 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:23:49.670392 140492 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } attempt: 3
I0227 17:23:49.675740 140624 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:23:49.679064 140492 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:23:49.679904 140492 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:43350: Network error: Client connection negotiation failed: client connection to 127.112.245.252:43350: connect: Connection refused (error 111)
I0227 17:23:49.680045 140624 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:23:49.680650 140624 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:43350: Network error: Client connection negotiation failed: client connection to 127.112.245.252:43350: connect: Connection refused (error 111)
I0227 17:23:49.699843 140624 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } attempt: 1
W0227 17:23:49.700775 140624 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:43350: Network error: Client connection negotiation failed: client connection to 127.112.245.252:43350: connect: Connection refused (error 111)
I0227 17:23:49.744066 140492 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } attempt: 1
W0227 17:23:49.744802 140492 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:43350: Network error: Client connection negotiation failed: client connection to 127.112.245.252:43350: connect: Connection refused (error 111)
I0227 17:23:49.748924 140624 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } attempt: 2
W0227 17:23:49.749639 140624 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:43350: Network error: Client connection negotiation failed: client connection to 127.112.245.252:43350: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:23:49.781098 140627 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:23:49.781379 140627 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:23:49.782742 140627 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:43740
W0227 17:23:49.782763 140627 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:23:49.782776 140627 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:23:49.782785 140627 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:23:49.782802 140627 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:23:49.785692 140627 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43740
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:43350
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.140627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:23:49.795858 140627 master_main.cc:80] Initializing master server...
I0227 17:23:49.797713 140627 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 165775us
I0227 17:23:49.799566 140627 fs_manager.cc:263] Metadata directory not provided
I0227 17:23:49.799638 140627 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:23:49.800009 140627 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:23:49.800050 140627 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:23:49.803321 140627 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/instance:
uuid: "e1c7902e23fd47719e2a87ffaee32a25"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.803791 140627 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal/instance:
uuid: "e1c7902e23fd47719e2a87ffaee32a25"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.813701 140627 fs_manager.cc:503] Time spent creating directory manager: real 0.010s	user 0.004s	sys 0.003s
I0227 17:23:49.814481 140627 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:23:49.814976 140627 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:23:49.817538 140627 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:23:49.817622 140627 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:23:49.819762 140627 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I0227 17:23:49.819859 140627 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "e1c7902e23fd47719e2a87ffaee32a25"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.819972 140627 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:23:49.823948 140492 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } attempt: 2
W0227 17:23:49.824631 140492 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:43350: Network error: Client connection negotiation failed: client connection to 127.112.245.252:43350: connect: Connection refused (error 111)
I0227 17:23:49.852582 140627 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:23:49.853649 140627 master_main.cc:83] Starting Master server...
I0227 17:23:49.861042 140624 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } attempt: 3
I0227 17:23:49.861534 140627 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.252:43350
I0227 17:23:49.862004 140627 webserver.cc:175] Starting webserver on 127.112.245.252:0
I0227 17:23:49.862028 140627 webserver.cc:186] Document root disabled
W0227 17:23:49.862176 140624 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:43350: Network error: Client connection negotiation failed: client connection to 127.112.245.252:43350: connect: Connection refused (error 111)
I0227 17:23:49.862727 140627 webserver.cc:313] Webserver started. Bound to: http://127.112.245.252:34890/
I0227 17:23:49.863431 140627 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
I0227 17:23:49.869827 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 140627
I0227 17:23:49.870239 140744 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:23:49.870882 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.112.245.193:0
--local_ip_for_outbound_sockets=127.112.245.193
--webserver_interface=127.112.245.193
--webserver_port=0
--tserver_master_addrs=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:23:49.880053 140744 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:23:49.889655 140744 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:23:49.895337 140744 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:23:49.904132 140744 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25: Bootstrap starting.
I0227 17:23:49.906745 140744 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25: No blocks or log segments found. Creating new log.
I0227 17:23:49.907842 140744 log.cc:525] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25: Log is configured to *not* fsync() on all Append() calls
I0227 17:23:49.910187 140744 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25: No bootstrap required, opened a new log
I0227 17:23:49.910778 140492 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } attempt: 3
I0227 17:23:49.916363 140744 raft_consensus.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:49.916484 140744 raft_consensus.cc:366] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:23:49.916555 140744 raft_consensus.cc:705] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c7902e23fd47719e2a87ffaee32a25, State: Initialized, Role: FOLLOWER
I0227 17:23:49.917277 140744 consensus_queue.cc:229] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:49.917557 140492 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: Bootstrap starting.
I0227 17:23:49.917968 140744 sys_catalog.cc:337] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.918036 140744 sys_catalog.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:49.918043 140759 sys_catalog.cc:337] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.918221 140759 sys_catalog.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:49.919668 140744 sys_catalog.cc:424] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:23:49.920657 140761 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0227 17:23:49.920722 140492 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: No blocks or log segments found. Creating new log.
I0227 17:23:49.921680 140627 master_main.cc:86] Master server successfully started.
I0227 17:23:49.922133 140492 log.cc:525] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: Log is configured to *not* fsync() on all Append() calls
W0227 17:23:49.923492 140762 catalog_manager.cc:1104] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:23:49.923532 140762 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:23:49.923549 140492 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: No bootstrap required, opened a new log
I0227 17:23:49.925529 140759 raft_consensus.cc:472] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:23:49.925812 140759 raft_consensus.cc:494] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:49.927151 140492 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:49.927227 140492 raft_consensus.cc:366] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:23:49.927281 140492 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f884ab0ecf045d2a5375ba08f7f4cd3, State: Initialized, Role: FOLLOWER
I0227 17:23:49.927706 140759 leader_election.cc:292] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), 56e0fd7bbdd64d88ac3bb214ee04ac3d (127.112.245.253:50162)
I0227 17:23:49.927768 140492 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:49.928175 140492 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.928213 140492 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:49.928220 140768 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.928244 140768 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:49.929301 140492 sys_catalog.cc:424] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:23:49.929832 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1c7902e23fd47719e2a87ffaee32a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:23:49.930063 140469 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1c7902e23fd47719e2a87ffaee32a25 in term 0.
I0227 17:23:49.930495 140773 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 9f884ab0ecf045d2a5375ba08f7f4cd3, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.930593 140242 master_main.cc:86] Master server successfully started.
I0227 17:23:49.930685 140688 leader_election.cc:306] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25; no voters: 
I0227 17:23:49.931018 140759 raft_consensus.cc:2640] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:23:49.931062 140759 raft_consensus.cc:472] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:23:49.931090 140759 raft_consensus.cc:2886] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:23:49.932806 140759 raft_consensus.cc:494] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:49.933347 140759 leader_election.cc:292] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [CANDIDATE]: Term 1 election: Requested vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), 56e0fd7bbdd64d88ac3bb214ee04ac3d (127.112.245.253:50162)
I0227 17:23:49.933693 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1c7902e23fd47719e2a87ffaee32a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3"
I0227 17:23:49.933763 140469 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 0 FOLLOWER]: Advancing to term 1
W0227 17:23:49.934928 140776 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:23:49.934979 140776 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:23:49.935158 140469 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1c7902e23fd47719e2a87ffaee32a25 in term 1.
I0227 17:23:49.935469 140688 leader_election.cc:306] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25; no voters: 
I0227 17:23:49.935637 140759 raft_consensus.cc:2640] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:23:49.935837 140759 raft_consensus.cc:667] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Becoming Leader. State: Replica: e1c7902e23fd47719e2a87ffaee32a25, State: Running, Role: LEADER
I0227 17:23:49.936219 140759 consensus_queue.cc:206] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:49.938685 140782 sys_catalog.cc:337] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1c7902e23fd47719e2a87ffaee32a25. Latest consensus state: current_term: 1 leader_uuid: "e1c7902e23fd47719e2a87ffaee32a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.938724 140782 sys_catalog.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: This master's current role is: LEADER
I0227 17:23:49.939186 140784 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:23:49.940459 140784 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
W0227 17:23:49.950829 140438 tablet.cc:1786] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:23:49.968714 140745 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:23:49.969123 140745 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:23:49.970353 140745 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:23:49.970521 140745 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.193
I0227 17:23:49.972658 140469 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Refusing update from remote peer e1c7902e23fd47719e2a87ffaee32a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:23:49.972911 140745 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.112.245.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.140745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.193
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:23:49.973942 140759 consensus_queue.cc:984] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:23:49.978539 140768 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1c7902e23fd47719e2a87ffaee32a25. Latest consensus state: current_term: 1 leader_uuid: "e1c7902e23fd47719e2a87ffaee32a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.978576 140768 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:49.980455 140759 sys_catalog.cc:337] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1c7902e23fd47719e2a87ffaee32a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.980489 140759 sys_catalog.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: This master's current role is: LEADER
I0227 17:23:49.981051 140768 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1c7902e23fd47719e2a87ffaee32a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:49.981091 140768 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:49.982897 140784 catalog_manager.cc:941] Generated new certificate authority record
I0227 17:23:49.982969 140745 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:23:49.984287 140784 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:23:49.984869 140745 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 165775us
I0227 17:23:49.985638 140745 fs_manager.cc:263] Metadata directory not provided
I0227 17:23:49.985735 140745 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:23:49.986063 140745 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:23:49.986130 140745 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:23:49.989194 140745 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/instance:
uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.989696 140745 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal/instance:
uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:49.998282 140784 catalog_manager.cc:4095] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25: Generated new TSK 0
I0227 17:23:49.998845 140784 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:23:49.999465 140745 fs_manager.cc:503] Time spent creating directory manager: real 0.009s	user 0.004s	sys 0.004s
I0227 17:23:50.000344 140745 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:23:50.000846 140745 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:23:50.004460 140745 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:23:50.004542 140745 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/data to 2721 blocks
I0227 17:23:50.006348 140745 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.000s
I0227 17:23:50.006460 140745 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3"
format_stamp: "Formatted at 2019-02-28 01:23:49 on ve0518.halxg.cloudera.com"
I0227 17:23:50.006574 140745 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:23:50.039317 140624 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } attempt: 4
I0227 17:23:50.046905 140624 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d: Bootstrap starting.
I0227 17:23:50.049587 140624 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d: No blocks or log segments found. Creating new log.
I0227 17:23:50.050894 140624 log.cc:525] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d: Log is configured to *not* fsync() on all Append() calls
I0227 17:23:50.052264 140624 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d: No bootstrap required, opened a new log
I0227 17:23:50.055790 140624 raft_consensus.cc:340] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:50.055862 140624 raft_consensus.cc:366] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:23:50.055912 140624 raft_consensus.cc:705] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e0fd7bbdd64d88ac3bb214ee04ac3d, State: Initialized, Role: FOLLOWER
I0227 17:23:50.056385 140624 consensus_queue.cc:229] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:50.056737 140624 sys_catalog.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:50.056767 140624 sys_catalog.cc:340] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:50.056941 140890 sys_catalog.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:50.056984 140890 sys_catalog.cc:340] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:50.057798 140624 sys_catalog.cc:424] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:23:50.057880 140601 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1c7902e23fd47719e2a87ffaee32a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" is_pre_election: true
I0227 17:23:50.058029 140599 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 0 FOLLOWER]: Advancing to term 1
I0227 17:23:50.058048 140600 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1c7902e23fd47719e2a87ffaee32a25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d"
I0227 17:23:50.059139 140495 master_main.cc:86] Master server successfully started.
I0227 17:23:50.059427 140599 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 1 FOLLOWER]: Refusing update from remote peer e1c7902e23fd47719e2a87ffaee32a25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:23:50.059620 140601 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1c7902e23fd47719e2a87ffaee32a25 for term 1 because replica is already servicing an update from a current leader or another vote.
I0227 17:23:50.059780 140600 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e1c7902e23fd47719e2a87ffaee32a25 for term 1 because replica is already servicing an update from a current leader or another vote.
I0227 17:23:50.060262 140782 consensus_queue.cc:984] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:23:50.061094 140892 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 56e0fd7bbdd64d88ac3bb214ee04ac3d, Raft Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
W0227 17:23:50.066720 140893 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:23:50.066789 140893 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:23:50.069275 140890 sys_catalog.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1c7902e23fd47719e2a87ffaee32a25. Latest consensus state: current_term: 1 leader_uuid: "e1c7902e23fd47719e2a87ffaee32a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:50.069304 140890 sys_catalog.cc:340] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:50.069401 140900 sys_catalog.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1c7902e23fd47719e2a87ffaee32a25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:50.069427 140900 sys_catalog.cc:340] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:50.077070 140745 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:23:50.078552 140745 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:23:50.078680 140745 tablet_server_main.cc:90] Starting tablet server...
I0227 17:23:50.090976 140745 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.193:35184
I0227 17:23:50.091471 140745 webserver.cc:175] Starting webserver on 127.112.245.193:0
I0227 17:23:50.091492 140745 webserver.cc:186] Document root disabled
I0227 17:23:50.092216 140745 webserver.cc:313] Webserver started. Bound to: http://127.112.245.193:39025/
I0227 17:23:50.092833 140745 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
I0227 17:23:50.094655 140745 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:23:50.103024 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 140745
I0227 17:23:50.103731 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data
--block_manager=log
--rpc_bind_addresses=127.112.245.194:0
--local_ip_for_outbound_sockets=127.112.245.194
--webserver_interface=127.112.245.194
--webserver_port=0
--tserver_master_addrs=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:23:50.107362 141014 heartbeater.cc:345] Connected to a master server at 127.112.245.252:43350
I0227 17:23:50.107470 141016 heartbeater.cc:345] Connected to a master server at 127.112.245.254:54091
I0227 17:23:50.107550 141016 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.107554 141014 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.107705 141015 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50162
I0227 17:23:50.107784 141015 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.108135 141016 heartbeater.cc:474] Master 127.112.245.254:54091 requested a full tablet report, sending...
I0227 17:23:50.108147 141015 heartbeater.cc:474] Master 127.112.245.253:50162 requested a full tablet report, sending...
I0227 17:23:50.108158 141014 heartbeater.cc:474] Master 127.112.245.252:43350 requested a full tablet report, sending...
I0227 17:23:50.109555 140459 ts_manager.cc:98] Registered new tserver with Master: 6d6e73bdfc8c4ce5afe6be3fa500a7c3 (127.112.245.193:35184)
I0227 17:23:50.109558 140704 ts_manager.cc:98] Registered new tserver with Master: 6d6e73bdfc8c4ce5afe6be3fa500a7c3 (127.112.245.193:35184)
I0227 17:23:50.109992 140591 ts_manager.cc:98] Registered new tserver with Master: 6d6e73bdfc8c4ce5afe6be3fa500a7c3 (127.112.245.193:35184)
I0227 17:23:50.110857 140704 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.193:45287
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:23:50.215590 141028 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:23:50.215921 141028 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:23:50.217293 141028 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:23:50.217393 141028 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.194
I0227 17:23:50.220203 141028 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.112.245.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.141028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.194
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:23:50.232332 141028 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:23:50.234689 141028 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 166275us
I0227 17:23:50.235652 141028 fs_manager.cc:263] Metadata directory not provided
I0227 17:23:50.235713 141028 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:23:50.236018 141028 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:23:50.236057 141028 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:23:50.239584 141028 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/instance:
uuid: "e0a2b4cb7c624e90bde59bcc09f2e804"
format_stamp: "Formatted at 2019-02-28 01:23:50 on ve0518.halxg.cloudera.com"
I0227 17:23:50.240157 141028 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal/instance:
uuid: "e0a2b4cb7c624e90bde59bcc09f2e804"
format_stamp: "Formatted at 2019-02-28 01:23:50 on ve0518.halxg.cloudera.com"
I0227 17:23:50.250521 141028 fs_manager.cc:503] Time spent creating directory manager: real 0.010s	user 0.006s	sys 0.003s
I0227 17:23:50.251562 141028 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:23:50.252266 141028 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:23:50.255965 141028 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:23:50.256103 141028 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/data to 2721 blocks
I0227 17:23:50.258195 141028 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I0227 17:23:50.258318 141028 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
uuid: "e0a2b4cb7c624e90bde59bcc09f2e804"
format_stamp: "Formatted at 2019-02-28 01:23:50 on ve0518.halxg.cloudera.com"
I0227 17:23:50.258414 141028 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:23:50.302354 141028 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:23:50.304008 141028 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:23:50.304174 141028 tablet_server_main.cc:90] Starting tablet server...
I0227 17:23:50.314551 141028 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.194:53979
I0227 17:23:50.315026 141028 webserver.cc:175] Starting webserver on 127.112.245.194:0
I0227 17:23:50.315090 141028 webserver.cc:186] Document root disabled
I0227 17:23:50.315799 141028 webserver.cc:313] Webserver started. Bound to: http://127.112.245.194:49089/
I0227 17:23:50.316402 141028 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
I0227 17:23:50.317991 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 141028
I0227 17:23:50.318099 141028 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:23:50.317131 141369 rpc_controller.cc:45] RpcController 0x7f0975f38c40 constructed
I0227 17:23:50.318773 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.112.245.195:0
--local_ip_for_outbound_sockets=127.112.245.195
--webserver_interface=127.112.245.195
--webserver_port=0
--tserver_master_addrs=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:23:50.328249 141369 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50162
I0227 17:23:50.328379 141369 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.328778 141370 heartbeater.cc:345] Connected to a master server at 127.112.245.254:54091
I0227 17:23:50.328783 141369 heartbeater.cc:474] Master 127.112.245.253:50162 requested a full tablet report, sending...
I0227 17:23:50.328807 141368 heartbeater.cc:345] Connected to a master server at 127.112.245.252:43350
I0227 17:23:50.328821 141370 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.328873 141368 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.328971 141370 heartbeater.cc:474] Master 127.112.245.254:54091 requested a full tablet report, sending...
I0227 17:23:50.329072 141368 heartbeater.cc:474] Master 127.112.245.252:43350 requested a full tablet report, sending...
I0227 17:23:50.329847 140704 ts_manager.cc:98] Registered new tserver with Master: e0a2b4cb7c624e90bde59bcc09f2e804 (127.112.245.194:53979)
I0227 17:23:50.329890 140459 ts_manager.cc:98] Registered new tserver with Master: e0a2b4cb7c624e90bde59bcc09f2e804 (127.112.245.194:53979)
I0227 17:23:50.329900 140591 ts_manager.cc:98] Registered new tserver with Master: e0a2b4cb7c624e90bde59bcc09f2e804 (127.112.245.194:53979)
I0227 17:23:50.330981 140704 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.194:32841
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:23:50.409656 141378 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:23:50.409924 141378 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:23:50.410977 141378 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:23:50.411063 141378 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.195
I0227 17:23:50.413218 141378 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.112.245.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:54091,127.112.245.253:50162,127.112.245.252:43350
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.141378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.195
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:23:50.424073 141378 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:23:50.425859 141378 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 166275us
I0227 17:23:50.426612 141378 fs_manager.cc:263] Metadata directory not provided
I0227 17:23:50.426694 141378 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:23:50.426977 141378 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:23:50.427034 141378 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:23:50.430034 141378 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/instance:
uuid: "bf3bc7b9fc694a91890569e39a9d5320"
format_stamp: "Formatted at 2019-02-28 01:23:50 on ve0518.halxg.cloudera.com"
I0227 17:23:50.430481 141378 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal/instance:
uuid: "bf3bc7b9fc694a91890569e39a9d5320"
format_stamp: "Formatted at 2019-02-28 01:23:50 on ve0518.halxg.cloudera.com"
I0227 17:23:50.438859 141378 fs_manager.cc:503] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.000s
I0227 17:23:50.439785 141378 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:23:50.440371 141378 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:23:50.444028 141378 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:23:50.444159 141378 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/data to 2721 blocks
I0227 17:23:50.445889 141378 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I0227 17:23:50.446066 141378 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
uuid: "bf3bc7b9fc694a91890569e39a9d5320"
format_stamp: "Formatted at 2019-02-28 01:23:50 on ve0518.halxg.cloudera.com"
I0227 17:23:50.446264 141378 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:23:50.480988 141378 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:23:50.482504 141378 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:23:50.482673 141378 tablet_server_main.cc:90] Starting tablet server...
I0227 17:23:50.494606 141378 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.195:56918
I0227 17:23:50.495074 141378 webserver.cc:175] Starting webserver on 127.112.245.195:0
I0227 17:23:50.495133 141378 webserver.cc:186] Document root disabled
I0227 17:23:50.495827 141378 webserver.cc:313] Webserver started. Bound to: http://127.112.245.195:50771/
I0227 17:23:50.496443 141378 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestCreateTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
I0227 17:23:50.497236 141378 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:23:50.506044 141610 heartbeater.cc:345] Connected to a master server at 127.112.245.252:43350
I0227 17:23:50.506165 141612 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50162
I0227 17:23:50.506186 141610 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.506217 141612 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.506305 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 141378
I0227 17:23:50.506695 141612 heartbeater.cc:474] Master 127.112.245.253:50162 requested a full tablet report, sending...
I0227 17:23:50.506696 141610 heartbeater.cc:474] Master 127.112.245.252:43350 requested a full tablet report, sending...
I0227 17:23:50.507591 141614 heartbeater.cc:345] Connected to a master server at 127.112.245.254:54091
I0227 17:23:50.507630 141614 heartbeater.cc:425] Registering TS with master...
I0227 17:23:50.507752 141614 heartbeater.cc:474] Master 127.112.245.254:54091 requested a full tablet report, sending...
I0227 17:23:50.507751 140591 ts_manager.cc:98] Registered new tserver with Master: bf3bc7b9fc694a91890569e39a9d5320 (127.112.245.195:56918)
I0227 17:23:50.508705 140459 ts_manager.cc:98] Registered new tserver with Master: bf3bc7b9fc694a91890569e39a9d5320 (127.112.245.195:56918)
I0227 17:23:50.508886 140704 ts_manager.cc:98] Registered new tserver with Master: bf3bc7b9fc694a91890569e39a9d5320 (127.112.245.195:56918)
I0227 17:23:50.509964 140704 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.195:55974
I0227 17:23:50.532349 115671 external_mini_cluster.cc:623] 3 TS(s) registered with all masters
I0227 17:23:50.548597 115671 master_failover-itest.cc:175] Pausing leader master
I0227 17:23:50.562885 140900 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1c7902e23fd47719e2a87ffaee32a25)
I0227 17:23:50.563050 140900 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:50.564182 140900 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:50.564661 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:23:50.564764 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:23:50.612862 141014 heartbeater.cc:466] Master 127.112.245.252:43350 was elected leader, sending a full tablet report...
I0227 17:23:50.832834 141368 heartbeater.cc:466] Master 127.112.245.252:43350 was elected leader, sending a full tablet report...
I0227 17:23:50.928896 142359 raft_consensus.cc:472] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1c7902e23fd47719e2a87ffaee32a25)
I0227 17:23:50.929204 142359 raft_consensus.cc:494] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:50.930549 142359 leader_election.cc:292] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e0fd7bbdd64d88ac3bb214ee04ac3d (127.112.245.253:50162), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:50.930999 140601 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" is_pre_election: true
I0227 17:23:50.931129 140601 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f884ab0ecf045d2a5375ba08f7f4cd3 in term 1.
I0227 17:23:50.931519 140297 leader_election.cc:306] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d, 9f884ab0ecf045d2a5375ba08f7f4cd3; no voters: 
I0227 17:23:50.932027 142359 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I0227 17:23:50.932060 142359 raft_consensus.cc:472] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1c7902e23fd47719e2a87ffaee32a25)
I0227 17:23:50.932082 142359 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:23:50.933472 142359 raft_consensus.cc:494] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:50.934193 142359 leader_election.cc:292] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [CANDIDATE]: Term 2 election: Requested vote from peers 56e0fd7bbdd64d88ac3bb214ee04ac3d (127.112.245.253:50162), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:50.934274 140601 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d"
I0227 17:23:50.934334 140601 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 1 FOLLOWER]: Advancing to term 2
I0227 17:23:50.935577 140601 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f884ab0ecf045d2a5375ba08f7f4cd3 in term 2.
I0227 17:23:50.935932 140297 leader_election.cc:306] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d, 9f884ab0ecf045d2a5375ba08f7f4cd3; no voters: 
I0227 17:23:50.936101 142359 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 FOLLOWER]: Leader election won for term 2
I0227 17:23:50.936256 142359 raft_consensus.cc:667] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Becoming Leader. State: Replica: 9f884ab0ecf045d2a5375ba08f7f4cd3, State: Running, Role: LEADER
I0227 17:23:50.936602 142359 consensus_queue.cc:206] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:50.937657 142397 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f884ab0ecf045d2a5375ba08f7f4cd3. Latest consensus state: current_term: 2 leader_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:50.937692 142397 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: This master's current role is: LEADER
I0227 17:23:50.938472 140776 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: acquiring CA information for follower catalog manager: success
I0227 17:23:50.939716 140776 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:23:50.939783 142402 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:23:50.940770 142402 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:23:50.941409 142402 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:23:50.941854 142402 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3: Loaded TSK: 0
I0227 17:23:50.942335 142402 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:23:51.011739 141610 heartbeater.cc:466] Master 127.112.245.252:43350 was elected leader, sending a full tablet report...
I0227 17:23:51.069532 140893 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d: acquiring CA information for follower catalog manager: success
I0227 17:23:51.070610 140893 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W0227 17:23:51.303553 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.738s (SENT)
I0227 17:23:51.303668 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:23:51.304316 142682 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0227 17:23:51.505858 140601 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Refusing update from remote peer 9f884ab0ecf045d2a5375ba08f7f4cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I0227 17:23:51.506747 142397 consensus_queue.cc:984] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I0227 17:23:51.508281 142682 sys_catalog.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f884ab0ecf045d2a5375ba08f7f4cd3. Latest consensus state: current_term: 2 leader_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:51.508322 142682 sys_catalog.cc:340] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:51.509675 142397 sys_catalog.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:51.509706 142397 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [sys.catalog]: This master's current role is: LEADER
I0227 17:23:51.509958 142682 sys_catalog.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:23:51.509985 142682 sys_catalog.cc:340] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:23:51.511343 141614 heartbeater.cc:466] Master 127.112.245.254:54091 was elected leader, sending a full tablet report...
W0227 17:23:51.584512 140299 leader_election.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.650s (SENT)
I0227 17:23:51.614154 141016 heartbeater.cc:466] Master 127.112.245.254:54091 was elected leader, sending a full tablet report...
W0227 17:23:51.614375 141014 heartbeater.cc:586] Failed to heartbeat to 127.112.245.252:43350 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.112.245.252:43350 timed out after 1.000s (SENT)
W0227 17:23:51.662289 140299 leader_election.cc:337] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.731s (SENT)
I0227 17:23:51.833977 141370 heartbeater.cc:466] Master 127.112.245.254:54091 was elected leader, sending a full tablet report...
W0227 17:23:51.834193 141368 heartbeater.cc:586] Failed to heartbeat to 127.112.245.252:43350 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.112.245.252:43350 timed out after 1.000s (SENT)
W0227 17:23:52.013550 141610 heartbeater.cc:586] Failed to heartbeat to 127.112.245.252:43350 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.112.245.252:43350 timed out after 1.000s (SENT)
I0227 17:23:54.136878 147149 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9f884ab0ecf045d2a5375ba08f7f4cd3)
I0227 17:23:54.137136 147149 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:54.137872 147149 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:54.138232 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:23:54.138304 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:23:54.692288 147420 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9f884ab0ecf045d2a5375ba08f7f4cd3)
I0227 17:23:54.692414 147420 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:54.692909 147420 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:54.693305 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:23:54.693392 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
W0227 17:23:54.701247 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.562s (SENT)
I0227 17:23:54.701314 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:23:54.701454 147420 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I0227 17:23:55.261453   936 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9f884ab0ecf045d2a5375ba08f7f4cd3)
I0227 17:23:55.261656   936 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:55.262403   936 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:55.262811 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:23:55.262934 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
W0227 17:23:55.385748 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.691s (SENT)
I0227 17:23:55.385825 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:23:55.385998   936 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W0227 17:23:55.873267 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.609s (SENT)
I0227 17:23:55.873384 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:23:55.873627   936 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I0227 17:23:56.716500  1737 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9f884ab0ecf045d2a5375ba08f7f4cd3)
I0227 17:23:56.716645  1737 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:56.717229  1737 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:56.719180 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:23:56.719274 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
W0227 17:23:57.225023 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.507s (SENT)
I0227 17:23:57.225106 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:23:57.225365  2746 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I0227 17:23:58.738072  4711 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9f884ab0ecf045d2a5375ba08f7f4cd3)
I0227 17:23:58.738289  4711 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:23:58.738783  4711 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:23:58.739387 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:23:58.739467 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
W0227 17:23:59.309278 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.570s (SENT)
I0227 17:23:59.309358 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:23:59.309689  5285 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W0227 17:24:00.567633 141633 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.252:43350)
I0227 17:24:00.750952 140459 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.254:57578:
name: "default.test_create_table_sync"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:24:00.752197 140459 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.test_create_table_sync in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:24:01.185582  6505 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9f884ab0ecf045d2a5375ba08f7f4cd3)
I0227 17:24:01.185708  6505 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:24:01.186130  6505 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:24:01.186628 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:24:01.186707 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:01.311650 140951 tablet_service.cc:774] Processing CreateTablet for tablet 8f7442f5253344898d808c1b5fb65c4a (table=default.test_create_table_sync [id=73f78bf683304da7bcf3997b818ca4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:24:01.312157 140951 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8f7442f5253344898d808c1b5fb65c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:24:01.312403 141302 tablet_service.cc:774] Processing CreateTablet for tablet 8f7442f5253344898d808c1b5fb65c4a (table=default.test_create_table_sync [id=73f78bf683304da7bcf3997b818ca4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:24:01.312913 141490 tablet_service.cc:774] Processing CreateTablet for tablet 8f7442f5253344898d808c1b5fb65c4a (table=default.test_create_table_sync [id=73f78bf683304da7bcf3997b818ca4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:24:01.313060 141302 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8f7442f5253344898d808c1b5fb65c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:24:01.313381 141490 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8f7442f5253344898d808c1b5fb65c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:24:01.318436 140951 ts_tablet_manager.cc:1175] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:24:01.318826  6723 ts_tablet_manager.cc:1030] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: Bootstrapping tablet
I0227 17:24:01.319169  6723 tablet_bootstrap.cc:439] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: Bootstrap starting.
I0227 17:24:01.321425  6723 tablet_bootstrap.cc:593] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: No blocks or log segments found. Creating new log.
I0227 17:24:01.322468 141302 ts_tablet_manager.cc:1175] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:24:01.322738  6723 log.cc:525] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: Log is configured to *not* fsync() on all Append() calls
I0227 17:24:01.322971  6728 ts_tablet_manager.cc:1030] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: Bootstrapping tablet
I0227 17:24:01.323225  6728 tablet_bootstrap.cc:439] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: Bootstrap starting.
I0227 17:24:01.323549 141490 ts_tablet_manager.cc:1175] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:24:01.324074  6733 ts_tablet_manager.cc:1030] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: Bootstrapping tablet
I0227 17:24:01.324116  6723 tablet_bootstrap.cc:439] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: No bootstrap required, opened a new log
I0227 17:24:01.324417  6723 ts_tablet_manager.cc:1047] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I0227 17:24:01.324529  6733 tablet_bootstrap.cc:439] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: Bootstrap starting.
I0227 17:24:01.325292  6728 tablet_bootstrap.cc:593] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: No blocks or log segments found. Creating new log.
I0227 17:24:01.326160  6728 log.cc:525] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: Log is configured to *not* fsync() on all Append() calls
I0227 17:24:01.327272  6728 tablet_bootstrap.cc:439] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: No bootstrap required, opened a new log
I0227 17:24:01.327417  6728 ts_tablet_manager.cc:1047] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I0227 17:24:01.327850  6733 tablet_bootstrap.cc:593] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: No blocks or log segments found. Creating new log.
I0227 17:24:01.327911  6723 raft_consensus.cc:340] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.327989  6723 raft_consensus.cc:366] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:24:01.328050  6723 raft_consensus.cc:705] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6e73bdfc8c4ce5afe6be3fa500a7c3, State: Initialized, Role: FOLLOWER
I0227 17:24:01.328460  6723 consensus_queue.cc:229] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.329293  6723 ts_tablet_manager.cc:1075] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.003s
I0227 17:24:01.330158  6733 log.cc:525] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: Log is configured to *not* fsync() on all Append() calls
I0227 17:24:01.330302  6728 raft_consensus.cc:340] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.330379  6728 raft_consensus.cc:366] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:24:01.330426  6728 raft_consensus.cc:705] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a2b4cb7c624e90bde59bcc09f2e804, State: Initialized, Role: FOLLOWER
I0227 17:24:01.330786  6728 consensus_queue.cc:229] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.331495  6728 ts_tablet_manager.cc:1075] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.001s
I0227 17:24:01.332484  6733 tablet_bootstrap.cc:439] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: No bootstrap required, opened a new log
I0227 17:24:01.332763  6733 ts_tablet_manager.cc:1047] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.001s
I0227 17:24:01.338495  6733 raft_consensus.cc:340] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.338647  6733 raft_consensus.cc:366] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:24:01.338747  6733 raft_consensus.cc:705] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3bc7b9fc694a91890569e39a9d5320, State: Initialized, Role: FOLLOWER
I0227 17:24:01.339604  6733 consensus_queue.cc:229] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.340241  6733 ts_tablet_manager.cc:1075] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
W0227 17:24:01.355662 141017 tablet.cc:1786] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:24:01.504818 141615 tablet.cc:1786] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:24:01.574641 141371 tablet.cc:1786] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:24:01.784950  6739 raft_consensus.cc:472] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:24:01.785197  6739 raft_consensus.cc:494] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.786860  6739 leader_election.cc:292] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d6e73bdfc8c4ce5afe6be3fa500a7c3 (127.112.245.193:35184), bf3bc7b9fc694a91890569e39a9d5320 (127.112.245.195:56918)
I0227 17:24:01.794767 141512 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8f7442f5253344898d808c1b5fb65c4a" candidate_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3bc7b9fc694a91890569e39a9d5320" is_pre_election: true
I0227 17:24:01.794860 140971 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8f7442f5253344898d808c1b5fb65c4a" candidate_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" is_pre_election: true
I0227 17:24:01.795007 141512 raft_consensus.cc:2346] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a2b4cb7c624e90bde59bcc09f2e804 in term 0.
I0227 17:24:01.795101 140971 raft_consensus.cc:2346] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a2b4cb7c624e90bde59bcc09f2e804 in term 0.
I0227 17:24:01.795817 141231 leader_election.cc:306] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf3bc7b9fc694a91890569e39a9d5320, e0a2b4cb7c624e90bde59bcc09f2e804; no voters: 
I0227 17:24:01.796108  6739 raft_consensus.cc:2640] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:24:01.796154  6739 raft_consensus.cc:472] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:24:01.796180  6739 raft_consensus.cc:2886] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:24:01.797492  6739 raft_consensus.cc:494] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.797914  6739 leader_election.cc:292] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [CANDIDATE]: Term 1 election: Requested vote from peers 6d6e73bdfc8c4ce5afe6be3fa500a7c3 (127.112.245.193:35184), bf3bc7b9fc694a91890569e39a9d5320 (127.112.245.195:56918)
I0227 17:24:01.798473 141512 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8f7442f5253344898d808c1b5fb65c4a" candidate_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3bc7b9fc694a91890569e39a9d5320"
I0227 17:24:01.798584 141512 raft_consensus.cc:2886] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:24:01.798614 140971 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8f7442f5253344898d808c1b5fb65c4a" candidate_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3"
I0227 17:24:01.798672 140971 raft_consensus.cc:2886] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:24:01.799966 140971 raft_consensus.cc:2346] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a2b4cb7c624e90bde59bcc09f2e804 in term 1.
I0227 17:24:01.800632 141229 leader_election.cc:306] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6e73bdfc8c4ce5afe6be3fa500a7c3, e0a2b4cb7c624e90bde59bcc09f2e804; no voters: 
I0227 17:24:01.800747  6739 raft_consensus.cc:2640] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:24:01.800777 141512 raft_consensus.cc:2346] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a2b4cb7c624e90bde59bcc09f2e804 in term 1.
I0227 17:24:01.800897  6739 raft_consensus.cc:667] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [term 1 LEADER]: Becoming Leader. State: Replica: e0a2b4cb7c624e90bde59bcc09f2e804, State: Running, Role: LEADER
I0227 17:24:01.801146  6739 consensus_queue.cc:206] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } }
I0227 17:24:01.801928  7908 raft_consensus.cc:472] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9f884ab0ecf045d2a5375ba08f7f4cd3)
I0227 17:24:01.802158  7908 raft_consensus.cc:494] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:24:01.803056  7908 leader_election.cc:292] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091), e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350)
I0227 17:24:01.803541 140469 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" is_pre_election: true
I0227 17:24:01.803614 140469 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:01.804725 140459 catalog_manager.cc:3791] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a2b4cb7c624e90bde59bcc09f2e804 (127.112.245.194). New cstate: current_term: 1 leader_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0a2b4cb7c624e90bde59bcc09f2e804" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 53979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 } health_report { overall_health: UNKNOWN } } }
W0227 17:24:01.875640 141278 tablet_service.cc:1915] Rejecting scan request for tablet 8f7442f5253344898d808c1b5fb65c4a: Uninitialized: safe time has not yet been initialized
W0227 17:24:01.892232 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.705s (SENT)
I0227 17:24:01.892298 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:24:01.892463  7908 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W0227 17:24:02.077772 141278 tablet_service.cc:1915] Rejecting scan request for tablet 8f7442f5253344898d808c1b5fb65c4a: Uninitialized: safe time has not yet been initialized
I0227 17:24:02.316967 140971 raft_consensus.cc:1181] T 8f7442f5253344898d808c1b5fb65c4a P 6d6e73bdfc8c4ce5afe6be3fa500a7c3 [term 1 FOLLOWER]: Refusing update from remote peer e0a2b4cb7c624e90bde59bcc09f2e804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:24:02.318131  8425 consensus_queue.cc:984] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d6e73bdfc8c4ce5afe6be3fa500a7c3" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 35184 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:24:02.325278 141512 raft_consensus.cc:1181] T 8f7442f5253344898d808c1b5fb65c4a P bf3bc7b9fc694a91890569e39a9d5320 [term 1 FOLLOWER]: Refusing update from remote peer e0a2b4cb7c624e90bde59bcc09f2e804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:24:02.326203  8431 consensus_queue.cc:984] T 8f7442f5253344898d808c1b5fb65c4a P e0a2b4cb7c624e90bde59bcc09f2e804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3bc7b9fc694a91890569e39a9d5320" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 56918 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0227 17:24:02.450008 140568 leader_election.cc:337] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1c7902e23fd47719e2a87ffaee32a25 (127.112.245.252:43350): Timed out: RequestConsensusVote RPC to 127.112.245.252:43350 timed out after 0.646s (SENT)
I0227 17:24:02.450153 140568 leader_election.cc:306] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e0fd7bbdd64d88ac3bb214ee04ac3d; no voters: 9f884ab0ecf045d2a5375ba08f7f4cd3, e1c7902e23fd47719e2a87ffaee32a25
I0227 17:24:02.450448  8777 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I0227 17:24:02.485528 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 140745
I0227 17:24:02.485677 140719 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.485762 140717 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.485908 140719 raft_consensus.cc:2307] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9f884ab0ecf045d2a5375ba08f7f4cd3 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:02.486160 140713 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.486203 140717 raft_consensus.cc:2321] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is already servicing an update from a current leader or another vote.
I0227 17:24:02.486297 140469 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [term 2 LEADER]: Rejecting Update request from peer e1c7902e23fd47719e2a87ffaee32a25 for earlier term 1. Current term is 2. Ops: []
I0227 17:24:02.486573 140719 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.486654 140719 raft_consensus.cc:2321] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is already servicing an update from a current leader or another vote.
I0227 17:24:02.487422 140714 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.487468 140713 raft_consensus.cc:2307] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:02.487526 140715 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.487542 140714 raft_consensus.cc:2321] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is already servicing an update from a current leader or another vote.
I0227 17:24:02.487651 140715 raft_consensus.cc:2321] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is already servicing an update from a current leader or another vote.
I0227 17:24:02.487677  8824 consensus_queue.cc:995] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 7, Time since last communication: 0.000s
I0227 17:24:02.487816 140710 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.487860 140710 raft_consensus.cc:2321] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is already servicing an update from a current leader or another vote.
I0227 17:24:02.487977 140716 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25"
I0227 17:24:02.488137 140711 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.488190  8824 consensus_queue.cc:1162] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [LEADER]: Peer 9f884ab0ecf045d2a5375ba08f7f4cd3 log is divergent from this leader: its last log entry 2.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I0227 17:24:02.488281 140718 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1c7902e23fd47719e2a87ffaee32a25" is_pre_election: true
I0227 17:24:02.488878 140716 raft_consensus.cc:2307] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 9f884ab0ecf045d2a5375ba08f7f4cd3 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:02.489166 140712 raft_consensus.cc:2881] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Stepping down as leader of term 1
I0227 17:24:02.489233 140712 raft_consensus.cc:705] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e1c7902e23fd47719e2a87ffaee32a25, State: Running, Role: LEADER
I0227 17:24:02.489601 140712 consensus_queue.cc:229] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } }
I0227 17:24:02.489683 140712 consensus_peers.cc:482] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 -> Peer 56e0fd7bbdd64d88ac3bb214ee04ac3d (127.112.245.253:50162): Closing peer: 56e0fd7bbdd64d88ac3bb214ee04ac3d
I0227 17:24:02.489754 140712 consensus_peers.cc:482] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 -> Peer 9f884ab0ecf045d2a5375ba08f7f4cd3 (127.112.245.254:54091): Closing peer: 9f884ab0ecf045d2a5375ba08f7f4cd3
I0227 17:24:02.489984 140712 raft_consensus.cc:2886] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:24:02.490459 140601 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 56e0fd7bbdd64d88ac3bb214ee04ac3d [term 2 FOLLOWER]: Rejecting Update request from peer e1c7902e23fd47719e2a87ffaee32a25 for earlier term 1. Current term is 2. Ops: []
I0227 17:24:02.491892 141610 heartbeater.cc:345] Connected to a master server at 127.112.245.252:43350
I0227 17:24:02.492012 141610 heartbeater.cc:466] Master 127.112.245.252:43350 was elected leader, sending a full tablet report...
I0227 17:24:02.492050 140712 raft_consensus.cc:1181] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 2 FOLLOWER]: Refusing update from remote peer 9f884ab0ecf045d2a5375ba08f7f4cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I0227 17:24:02.492202 140718 raft_consensus.cc:2321] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 2 because replica is already servicing an update from a current leader or another vote.
I0227 17:24:02.492265 141368 heartbeater.cc:345] Connected to a master server at 127.112.245.252:43350
I0227 17:24:02.492367 140711 raft_consensus.cc:2321] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e0fd7bbdd64d88ac3bb214ee04ac3d for term 3 because replica is already servicing an update from a current leader or another vote.
I0227 17:24:02.492385 141368 heartbeater.cc:466] Master 127.112.245.252:43350 was elected leader, sending a full tablet report...
I0227 17:24:02.492597  8823 sys_catalog.cc:337] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:24:02.492632  8823 sys_catalog.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:02.493010  8482 consensus_queue.cc:984] T 00000000000000000000000000000000 P 9f884ab0ecf045d2a5375ba08f7f4cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I0227 17:24:02.497031  8824 sys_catalog.cc:337] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:24:02.497066  8824 sys_catalog.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:02.497213  8823 sys_catalog.cc:337] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f884ab0ecf045d2a5375ba08f7f4cd3. Latest consensus state: current_term: 2 leader_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f884ab0ecf045d2a5375ba08f7f4cd3" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 54091 } } peers { permanent_uuid: "56e0fd7bbdd64d88ac3bb214ee04ac3d" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50162 } } peers { permanent_uuid: "e1c7902e23fd47719e2a87ffaee32a25" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 43350 } } }
I0227 17:24:02.497272  8823 sys_catalog.cc:340] T 00000000000000000000000000000000 P e1c7902e23fd47719e2a87ffaee32a25 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:02.497771  8835 mvcc.cc:203] Tried to move safe_time back from 6354194603238793216 to 6354194558718947328. Current Snapshot: MvccSnapshot[committed={T|T < 6354194603238793216}]
W0227 17:24:02.500108 140296 connection.cc:647] server connection from 127.112.245.193:51627 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:51627): Connection reset by peer (error 104)
W0227 17:24:02.500180 140688 connection.cc:647] server connection from 127.112.245.193:45287 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0227 17:24:02.500228 141229 connection.cc:647] client connection to 127.112.245.193:35184 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:35184): Connection reset by peer (error 104)
W0227 17:24:02.500188 140568 connection.cc:647] server connection from 127.112.245.193:56440 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:56440): Connection reset by peer (error 104)
W0227 17:24:02.500545 140688 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Broken pipe
I0227 17:24:02.501160 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 141028
W0227 17:24:02.517056 140689 connection.cc:647] server connection from 127.112.245.194:32841 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:32841): Connection reset by peer (error 104)
W0227 17:24:02.517109 140565 connection.cc:647] server connection from 127.112.245.194:52838 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:52838): Connection reset by peer (error 104)
W0227 17:24:02.517117 140299 connection.cc:647] server connection from 127.112.245.194:33983 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:33983): Connection reset by peer (error 104)
W0227 17:24:02.517328 141426 connection.cc:647] server connection from 127.112.245.194:37980 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:37980): Connection reset by peer (error 104)
I0227 17:24:02.517738 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 141378
W0227 17:24:02.533097 140689 connection.cc:647] server connection from 127.112.245.195:55974 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:55974): Connection reset by peer (error 104)
W0227 17:24:02.533107 140296 connection.cc:647] server connection from 127.112.245.195:46138 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:46138): Connection reset by peer (error 104)
W0227 17:24:02.533120 140567 connection.cc:647] server connection from 127.112.245.195:48187 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:48187): Connection reset by peer (error 104)
I0227 17:24:02.533437 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 140242
I0227 17:24:02.559494 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 140495
I0227 17:24:02.577550 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 140627
Shutting down hive metastore.
[       OK ] HmsConfigurations/MasterFailoverTest.TestCreateTableSync/1 (32729 ms)
[ RUN      ] HmsConfigurations/MasterFailoverTest.TestDeleteTableSync/1
2019-02-27 17:24:04: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1
log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hive-498021fa15186aee8b282d3c032fbd2cede6bec4/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2403 T1 oahu.NativeCodeLoader.<clinit> WARN Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting hive metastore on port 0
2918 T1 odu.Log4JLogger.info Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
2919 T1 odu.Log4JLogger.info Property datanucleus.cache.level2 unknown - will be ignored
I0227 17:24:23.187819 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:39035
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--hive_metastore_uris=thrift://127.0.0.1:47540 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:24:23.369711 34769 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:24:23.370149 34769 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:24:23.371500 34769 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:47540
W0227 17:24:23.371590 34769 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:24:23.371673 34769 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:24:23.371748 34769 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:24:23.371829 34769 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:24:23.375164 34769 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:47540
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.34769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:24:23.390372 34769 master_main.cc:80] Initializing master server...
I0227 17:24:23.391412 34769 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 182775us
I0227 17:24:23.393141 34769 fs_manager.cc:263] Metadata directory not provided
I0227 17:24:23.393254 34769 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:24:23.393628 34769 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:24:23.393709 34769 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:24:23.396948 34769 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/instance:
uuid: "45d749848fc844e1af0e5b15b810994f"
format_stamp: "Formatted at 2019-02-28 01:24:23 on ve0518.halxg.cloudera.com"
I0227 17:24:23.397437 34769 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal/instance:
uuid: "45d749848fc844e1af0e5b15b810994f"
format_stamp: "Formatted at 2019-02-28 01:24:23 on ve0518.halxg.cloudera.com"
I0227 17:24:23.412742 34769 fs_manager.cc:503] Time spent creating directory manager: real 0.015s	user 0.005s	sys 0.003s
I0227 17:24:23.413857 34769 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:24:23.414578 34769 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:24:23.420387 34769 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:24:23.420562 34769 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/data to 2721 blocks
I0227 17:24:23.422878 34769 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.001s
I0227 17:24:23.423040 34769 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
uuid: "45d749848fc844e1af0e5b15b810994f"
format_stamp: "Formatted at 2019-02-28 01:24:23 on ve0518.halxg.cloudera.com"
I0227 17:24:23.423219 34769 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:24:23.465617 34769 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:24:23.466681 34769 master_main.cc:83] Starting Master server...
I0227 17:24:23.480783 34769 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.254:39035
I0227 17:24:23.481302 34769 webserver.cc:175] Starting webserver on 127.112.245.254:0
I0227 17:24:23.481325 34769 webserver.cc:186] Document root disabled
I0227 17:24:23.482048 34769 webserver.cc:313] Webserver started. Bound to: http://127.112.245.254:57433/
I0227 17:24:23.482720 34769 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
I0227 17:24:23.485960 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 34769
I0227 17:24:23.486573 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:41272
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--hive_metastore_uris=thrift://127.0.0.1:47540 with env {}
I0227 17:24:23.504719 35154 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:24:23.515087 35154 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:24:23.533035 35154 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:24:23.537670 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:41272: Network error: Client connection negotiation failed: client connection to 127.112.245.253:41272: connect: Connection refused (error 111)
I0227 17:24:23.557215 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } attempt: 1
W0227 17:24:23.557989 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:41272: Network error: Client connection negotiation failed: client connection to 127.112.245.253:41272: connect: Connection refused (error 111)
I0227 17:24:23.606319 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } attempt: 2
W0227 17:24:23.607442 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:41272: Network error: Client connection negotiation failed: client connection to 127.112.245.253:41272: connect: Connection refused (error 111)
I0227 17:24:23.718726 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } attempt: 3
W0227 17:24:23.719878 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:41272: Network error: Client connection negotiation failed: client connection to 127.112.245.253:41272: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:24:23.729077 35155 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:24:23.729359 35155 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:24:23.733408 35155 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:47540
W0227 17:24:23.733516 35155 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:24:23.733600 35155 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:24:23.733677 35155 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:24:23.733793 35155 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:24:23.739120 35155 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:47540
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:41272
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.35155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:24:23.778553 35155 master_main.cc:80] Initializing master server...
I0227 17:24:23.781392 35155 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 182775us
I0227 17:24:23.785428 35155 fs_manager.cc:263] Metadata directory not provided
I0227 17:24:23.785631 35155 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:24:23.786007 35155 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:24:23.786320 35155 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:24:23.794680 35155 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/instance:
uuid: "bbd34c4935164b99889130f50a1ac965"
format_stamp: "Formatted at 2019-02-28 01:24:23 on ve0518.halxg.cloudera.com"
I0227 17:24:23.796142 35155 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal/instance:
uuid: "bbd34c4935164b99889130f50a1ac965"
format_stamp: "Formatted at 2019-02-28 01:24:23 on ve0518.halxg.cloudera.com"
I0227 17:24:23.836042 35155 fs_manager.cc:503] Time spent creating directory manager: real 0.038s	user 0.007s	sys 0.002s
I0227 17:24:23.837409 35155 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:24:23.838853 35155 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:24:23.842551 35155 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:24:23.843689 35155 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/data to 2721 blocks
I0227 17:24:23.846027 35155 fs_manager.cc:419] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.000s
I0227 17:24:23.846184 35155 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
uuid: "bbd34c4935164b99889130f50a1ac965"
format_stamp: "Formatted at 2019-02-28 01:24:23 on ve0518.halxg.cloudera.com"
I0227 17:24:23.846386 35155 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:24:23.896015 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } attempt: 4
W0227 17:24:23.896792 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:41272: Network error: Client connection negotiation failed: client connection to 127.112.245.253:41272: connect: Connection refused (error 111)
I0227 17:24:23.929510 35155 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:24:23.930773 35155 master_main.cc:83] Starting Master server...
I0227 17:24:23.937979 35155 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.253:41272
I0227 17:24:23.938666 35155 webserver.cc:175] Starting webserver on 127.112.245.253:0
I0227 17:24:23.938745 35155 webserver.cc:186] Document root disabled
I0227 17:24:23.939580 35155 webserver.cc:313] Webserver started. Bound to: http://127.112.245.253:48364/
I0227 17:24:23.940366 35155 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
I0227 17:24:23.949717 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 35155
I0227 17:24:23.950362 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:34480
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--hive_metastore_uris=thrift://127.0.0.1:47540 with env {}
I0227 17:24:23.952214 35579 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:24:23.962688 35579 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:24:24.004681 35579 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:24:24.022418 35579 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:24:24.023157 35579 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.042377 35579 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 1
W0227 17:24:24.043072 35579 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.091223 35579 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 2
W0227 17:24:24.091846 35579 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.199920 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } attempt: 5
I0227 17:24:24.202989 35579 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 3
W0227 17:24:24.203644 35579 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.205554 35154 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:24:24.206188 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:24:24.210413 35585 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:24:24.217607 35585 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:24:24.219219 35585 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:47540
W0227 17:24:24.219285 35585 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:24:24.219344 35585 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:24:24.219396 35585 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:24:24.219458 35585 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:24:24.228375 35585 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:47540
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:34480
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.35585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:24:24.244315 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 1
I0227 17:24:24.244719 35585 master_main.cc:80] Initializing master server...
W0227 17:24:24.245281 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.245717 35585 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 183275us
I0227 17:24:24.251906 35585 fs_manager.cc:263] Metadata directory not provided
I0227 17:24:24.257577 35585 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:24:24.258150 35585 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:24:24.258258 35585 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:24:24.261621 35585 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/instance:
uuid: "eaae685ab2234943bccdfa4e9d6b87e1"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:24.262156 35585 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal/instance:
uuid: "eaae685ab2234943bccdfa4e9d6b87e1"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:24.291368 35585 fs_manager.cc:503] Time spent creating directory manager: real 0.024s	user 0.005s	sys 0.002s
I0227 17:24:24.292302 35585 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:24:24.292881 35585 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:24:24.294054 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 2
W0227 17:24:24.294811 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.295936 35585 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:24:24.296034 35585 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:24:24.298323 35585 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I0227 17:24:24.298568 35585 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "eaae685ab2234943bccdfa4e9d6b87e1"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:24.298974 35585 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:24:24.379798 35579 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 4
W0227 17:24:24.380925 35579 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.394937 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 3
W0227 17:24:24.395738 35154 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:34480: Network error: Client connection negotiation failed: client connection to 127.112.245.252:34480: connect: Connection refused (error 111)
I0227 17:24:24.413923 35585 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:24:24.416934 35585 master_main.cc:83] Starting Master server...
I0227 17:24:24.426805 35585 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.252:34480
I0227 17:24:24.427261 35585 webserver.cc:175] Starting webserver on 127.112.245.252:0
I0227 17:24:24.427287 35585 webserver.cc:186] Document root disabled
I0227 17:24:24.436760 35585 webserver.cc:313] Webserver started. Bound to: http://127.112.245.252:48737/
I0227 17:24:24.437482 35585 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
I0227 17:24:24.440649 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 35585
I0227 17:24:24.441366 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.112.245.193:0
--local_ip_for_outbound_sockets=127.112.245.193
--webserver_interface=127.112.245.193
--webserver_port=0
--tserver_master_addrs=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:24:24.453238 36158 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:24:24.459235 36158 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:24:24.499790 36158 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:24:24.536181 35154 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 4
I0227 17:24:24.553077 36158 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:24:24.573918 36158 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: Bootstrap starting.
I0227 17:24:24.574524 35154 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: Bootstrap starting.
I0227 17:24:24.576627 35154 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: No blocks or log segments found. Creating new log.
I0227 17:24:24.576630 36158 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: No blocks or log segments found. Creating new log.
I0227 17:24:24.577754 36158 log.cc:525] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: Log is configured to *not* fsync() on all Append() calls
I0227 17:24:24.578168 35154 log.cc:525] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: Log is configured to *not* fsync() on all Append() calls
I0227 17:24:24.586678 36158 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: No bootstrap required, opened a new log
I0227 17:24:24.589278 35154 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: No bootstrap required, opened a new log
I0227 17:24:24.590683 36158 raft_consensus.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.590836 36158 raft_consensus.cc:366] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:24:24.590960 36158 raft_consensus.cc:705] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaae685ab2234943bccdfa4e9d6b87e1, State: Initialized, Role: FOLLOWER
I0227 17:24:24.591562 36158 consensus_queue.cc:229] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.591984 36158 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.592084 36158 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.592345 35154 raft_consensus.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.592417 35154 raft_consensus.cc:366] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:24:24.592468 35154 raft_consensus.cc:705] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d749848fc844e1af0e5b15b810994f, State: Initialized, Role: FOLLOWER
I0227 17:24:24.593466 36158 sys_catalog.cc:424] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:24:24.593801 35154 consensus_queue.cc:229] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.594339 35154 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.594373 35154 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.595765 35154 sys_catalog.cc:424] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:24:24.597259 34769 master_main.cc:86] Master server successfully started.
I0227 17:24:24.600826 35585 master_main.cc:86] Master server successfully started.
I0227 17:24:24.599261 36193 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: eaae685ab2234943bccdfa4e9d6b87e1, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.602974 36191 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.603087 36191 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:24:24.604171 36197 catalog_manager.cc:1104] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:24:24.604202 36197 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:24:24.605779 36195 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 45d749848fc844e1af0e5b15b810994f, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.600211 36190 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.607861 36190 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:24:24.610585 36196 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:24:24.610698 36196 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:24:24.641332 36160 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:24:24.649015 36160 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:24:24.650483 36160 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:24:24.650657 36160 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.193
I0227 17:24:24.653007 36160 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.112.245.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.36160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.193
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
W0227 17:24:24.672358 36075 tablet.cc:1786] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:24:24.684175 35579 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } attempt: 5
I0227 17:24:24.684685 36160 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:24:24.686800 36160 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 183275us
I0227 17:24:24.687594 36160 fs_manager.cc:263] Metadata directory not provided
I0227 17:24:24.687688 36160 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:24:24.688143 36160 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:24:24.689872 36160 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:24:24.690848 35579 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965: Bootstrap starting.
I0227 17:24:24.692880 36160 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/instance:
uuid: "abb7bc57957e4f9c8a5779ebe08a05c7"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:24.694697 35579 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965: No blocks or log segments found. Creating new log.
I0227 17:24:24.696097 36160 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal/instance:
uuid: "abb7bc57957e4f9c8a5779ebe08a05c7"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:24.697018 35579 log.cc:525] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965: Log is configured to *not* fsync() on all Append() calls
I0227 17:24:24.707962 35579 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965: No bootstrap required, opened a new log
I0227 17:24:24.711750 35579 raft_consensus.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.711841 35579 raft_consensus.cc:366] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:24:24.711896 35579 raft_consensus.cc:705] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd34c4935164b99889130f50a1ac965, State: Initialized, Role: FOLLOWER
I0227 17:24:24.712404 35579 consensus_queue.cc:229] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.712821 35579 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.712858 35579 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.713919 35579 sys_catalog.cc:424] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:24:24.716506 35155 master_main.cc:86] Master server successfully started.
W0227 17:24:24.717855 35109 tablet.cc:1786] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:24:24.723374 36223 catalog_manager.cc:1104] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:24:24.723418 36223 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:24:24.727249 36219 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.727325 36219 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.730886 36160 fs_manager.cc:503] Time spent creating directory manager: real 0.034s	user 0.005s	sys 0.001s
I0227 17:24:24.731762 36160 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:24:24.732286 36160 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:24:24.734206 36222 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: bbd34c4935164b99889130f50a1ac965, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.735251 36160 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:24:24.735816 36160 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/data to 2721 blocks
I0227 17:24:24.736191 36190 raft_consensus.cc:472] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:24:24.736364 36190 raft_consensus.cc:494] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.737601 36190 leader_election.cc:292] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272)
I0227 17:24:24.737649 36160 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.001s
I0227 17:24:24.737758 36160 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
uuid: "abb7bc57957e4f9c8a5779ebe08a05c7"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:24.738121 36160 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:24:24.738171 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:24.738371 35555 raft_consensus.cc:2346] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaae685ab2234943bccdfa4e9d6b87e1 in term 0.
I0227 17:24:24.738504 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:24.738759 35126 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaae685ab2234943bccdfa4e9d6b87e1 in term 0.
I0227 17:24:24.738883 35936 leader_election.cc:306] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1; no voters: 
I0227 17:24:24.739179 36190 raft_consensus.cc:2640] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:24:24.739226 36190 raft_consensus.cc:472] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:24:24.739255 36190 raft_consensus.cc:2886] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:24:24.749752 36190 raft_consensus.cc:494] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.750339 36190 leader_election.cc:292] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 1 election: Requested vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272)
I0227 17:24:24.751096 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f"
I0227 17:24:24.751204 35126 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 0 FOLLOWER]: Advancing to term 1
I0227 17:24:24.752588 35126 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaae685ab2234943bccdfa4e9d6b87e1 in term 1.
I0227 17:24:24.753206 35937 leader_election.cc:306] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1; no voters: 
I0227 17:24:24.753370 36190 raft_consensus.cc:2640] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:24:24.753605 36190 raft_consensus.cc:667] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 LEADER]: Becoming Leader. State: Replica: eaae685ab2234943bccdfa4e9d6b87e1, State: Running, Role: LEADER
I0227 17:24:24.754034 36190 consensus_queue.cc:206] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:24.754406 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965"
I0227 17:24:24.754482 35555 raft_consensus.cc:2886] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:24:24.755647 35555 raft_consensus.cc:2346] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaae685ab2234943bccdfa4e9d6b87e1 in term 1.
I0227 17:24:24.755709 36190 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaae685ab2234943bccdfa4e9d6b87e1. Latest consensus state: current_term: 1 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.755738 36190 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:24.767554 36239 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:24:24.770604 36239 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:24:24.794126 36160 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:24:24.797881 36160 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:24:24.798105 36160 tablet_server_main.cc:90] Starting tablet server...
I0227 17:24:24.820649 36160 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.193:36713
I0227 17:24:24.821007 36160 webserver.cc:175] Starting webserver on 127.112.245.193:0
I0227 17:24:24.821032 36160 webserver.cc:186] Document root disabled
I0227 17:24:24.821662 36160 webserver.cc:313] Webserver started. Bound to: http://127.112.245.193:54821/
I0227 17:24:24.822211 36160 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
I0227 17:24:24.823202 36160 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:24:24.828130 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 36160
I0227 17:24:24.828749 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data
--block_manager=log
--rpc_bind_addresses=127.112.245.194:0
--local_ip_for_outbound_sockets=127.112.245.194
--webserver_interface=127.112.245.194
--webserver_port=0
--tserver_master_addrs=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:24:24.845727 35555 raft_consensus.cc:1181] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 1 FOLLOWER]: Refusing update from remote peer eaae685ab2234943bccdfa4e9d6b87e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:24:24.846475 36237 consensus_queue.cc:984] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:24:24.847200 35126 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 1 FOLLOWER]: Refusing update from remote peer eaae685ab2234943bccdfa4e9d6b87e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:24:24.847829 36237 consensus_queue.cc:984] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:24:24.849858 36219 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaae685ab2234943bccdfa4e9d6b87e1. Latest consensus state: current_term: 1 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.849889 36219 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.856771 36409 heartbeater.cc:345] Connected to a master server at 127.112.245.254:39035
I0227 17:24:24.856771 36407 heartbeater.cc:345] Connected to a master server at 127.112.245.252:34480
I0227 17:24:24.856936 36409 heartbeater.cc:425] Registering TS with master...
I0227 17:24:24.856936 36407 heartbeater.cc:425] Registering TS with master...
I0227 17:24:24.857440 36407 heartbeater.cc:474] Master 127.112.245.252:34480 requested a full tablet report, sending...
I0227 17:24:24.857439 36409 heartbeater.cc:474] Master 127.112.245.254:39035 requested a full tablet report, sending...
I0227 17:24:24.858758 36091 ts_manager.cc:98] Registered new tserver with Master: abb7bc57957e4f9c8a5779ebe08a05c7 (127.112.245.193:36713)
I0227 17:24:24.860060 35117 ts_manager.cc:98] Registered new tserver with Master: abb7bc57957e4f9c8a5779ebe08a05c7 (127.112.245.193:36713)
I0227 17:24:24.860065 36191 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaae685ab2234943bccdfa4e9d6b87e1. Latest consensus state: current_term: 1 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.860601 36191 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.862828 36408 heartbeater.cc:345] Connected to a master server at 127.112.245.253:41272
I0227 17:24:24.862879 36408 heartbeater.cc:425] Registering TS with master...
I0227 17:24:24.863061 36408 heartbeater.cc:474] Master 127.112.245.253:41272 requested a full tablet report, sending...
I0227 17:24:24.864204 35537 ts_manager.cc:98] Registered new tserver with Master: abb7bc57957e4f9c8a5779ebe08a05c7 (127.112.245.193:36713)
I0227 17:24:24.864785 36237 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.864818 36237 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:24.865309 36237 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.865334 36237 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:24.866240 36191 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.866271 36191 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.866329 36219 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:24.866356 36219 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:24.874339 36239 catalog_manager.cc:941] Generated new certificate authority record
I0227 17:24:24.875815 36239 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:24:24.889490 36239 catalog_manager.cc:4095] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: Generated new TSK 0
I0227 17:24:24.890120 36239 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:24:24.978957 36413 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:24:24.979385 36413 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:24:24.980551 36413 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:24:24.980705 36413 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.194
I0227 17:24:24.982949 36413 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.112.245.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.36413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.194
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:24:24.993484 36413 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:24:24.995474 36413 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 183275us
I0227 17:24:24.996330 36413 fs_manager.cc:263] Metadata directory not provided
I0227 17:24:24.996446 36413 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:24:24.996786 36413 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:24:24.996870 36413 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:24:25.000355 36413 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/instance:
uuid: "6f41fa4ee2ac472298b75a76a919e961"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:25.001427 36413 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal/instance:
uuid: "6f41fa4ee2ac472298b75a76a919e961"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:25.025794 36413 fs_manager.cc:503] Time spent creating directory manager: real 0.024s	user 0.007s	sys 0.001s
I0227 17:24:25.027104 36413 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:24:25.027707 36413 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:24:25.033392 36413 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:24:25.033601 36413 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/data to 2721 blocks
I0227 17:24:25.036615 36413 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.000s
I0227 17:24:25.036746 36413 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
uuid: "6f41fa4ee2ac472298b75a76a919e961"
format_stamp: "Formatted at 2019-02-28 01:24:24 on ve0518.halxg.cloudera.com"
I0227 17:24:25.036875 36413 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:24:25.081418 36413 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:24:25.082995 36413 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:24:25.083218 36413 tablet_server_main.cc:90] Starting tablet server...
I0227 17:24:25.102512 36413 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.194:47937
I0227 17:24:25.102885 36413 webserver.cc:175] Starting webserver on 127.112.245.194:0
I0227 17:24:25.102898 36413 webserver.cc:186] Document root disabled
I0227 17:24:25.105163 36413 webserver.cc:313] Webserver started. Bound to: http://127.112.245.194:37074/
I0227 17:24:25.105705 36413 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
I0227 17:24:25.108463 36413 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:24:25.111639 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 36413
I0227 17:24:25.112310 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.112.245.195:0
--local_ip_for_outbound_sockets=127.112.245.195
--webserver_interface=127.112.245.195
--webserver_port=0
--tserver_master_addrs=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:24:25.137828 36745 heartbeater.cc:345] Connected to a master server at 127.112.245.252:34480
I0227 17:24:25.137961 36745 heartbeater.cc:425] Registering TS with master...
I0227 17:24:25.141636 36747 heartbeater.cc:345] Connected to a master server at 127.112.245.254:39035
I0227 17:24:25.141693 36747 heartbeater.cc:425] Registering TS with master...
I0227 17:24:25.142204 36747 heartbeater.cc:474] Master 127.112.245.254:39035 requested a full tablet report, sending...
I0227 17:24:25.142885 36745 heartbeater.cc:474] Master 127.112.245.252:34480 requested a full tablet report, sending...
I0227 17:24:25.143296 35117 ts_manager.cc:98] Registered new tserver with Master: 6f41fa4ee2ac472298b75a76a919e961 (127.112.245.194:47937)
I0227 17:24:25.142933 36746 heartbeater.cc:345] Connected to a master server at 127.112.245.253:41272
I0227 17:24:25.146215 36746 heartbeater.cc:425] Registering TS with master...
I0227 17:24:25.146385 36746 heartbeater.cc:474] Master 127.112.245.253:41272 requested a full tablet report, sending...
I0227 17:24:25.146628 36091 ts_manager.cc:98] Registered new tserver with Master: 6f41fa4ee2ac472298b75a76a919e961 (127.112.245.194:47937)
I0227 17:24:25.147233 35537 ts_manager.cc:98] Registered new tserver with Master: 6f41fa4ee2ac472298b75a76a919e961 (127.112.245.194:47937)
I0227 17:24:25.148423 36091 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.194:47912
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:24:25.334384 36758 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:24:25.334695 36758 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:24:25.335772 36758 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:24:25.339258 36758 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.195
I0227 17:24:25.341665 36758 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.112.245.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:39035,127.112.245.253:41272,127.112.245.252:34480
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.36758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.195
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:24:25.361601 36091 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.193:41840
I0227 17:24:25.364621 36758 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:24:25.368099 36758 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 183775us
I0227 17:24:25.370419 36758 fs_manager.cc:263] Metadata directory not provided
I0227 17:24:25.370579 36758 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:24:25.370937 36758 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:24:25.371021 36758 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:24:25.378561 36758 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/instance:
uuid: "a9929d795dd545d6ad6ba5918488ad2f"
format_stamp: "Formatted at 2019-02-28 01:24:25 on ve0518.halxg.cloudera.com"
I0227 17:24:25.386570 36758 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal/instance:
uuid: "a9929d795dd545d6ad6ba5918488ad2f"
format_stamp: "Formatted at 2019-02-28 01:24:25 on ve0518.halxg.cloudera.com"
I0227 17:24:25.396404 36903 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:25.396536 36903 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:25.399035 36903 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:25.399368 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:25.399453 35126 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbd34c4935164b99889130f50a1ac965 in term 1.
I0227 17:24:25.399583 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:25.399749 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:25.399838 35507 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f, bbd34c4935164b99889130f50a1ac965; no voters: 
I0227 17:24:25.400182 36903 raft_consensus.cc:2640] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 1 FOLLOWER]: Leader pre-election won for term 2
I0227 17:24:25.400240 36903 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:25.400300 36903 raft_consensus.cc:2886] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:24:25.401520 36903 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:25.405943 36903 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 2 election: Requested vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:25.406119 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f"
I0227 17:24:25.406185 35126 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 1 FOLLOWER]: Advancing to term 2
I0227 17:24:25.407212 35126 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbd34c4935164b99889130f50a1ac965 in term 2.
I0227 17:24:25.407549 35507 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f, bbd34c4935164b99889130f50a1ac965; no voters: 
I0227 17:24:25.407753 36903 raft_consensus.cc:2640] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 FOLLOWER]: Leader election won for term 2
I0227 17:24:25.407791 36758 fs_manager.cc:503] Time spent creating directory manager: real 0.021s	user 0.005s	sys 0.002s
I0227 17:24:25.407891 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1"
I0227 17:24:25.407968 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 LEADER]: Leader election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:25.408118 36903 raft_consensus.cc:667] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 LEADER]: Becoming Leader. State: Replica: bbd34c4935164b99889130f50a1ac965, State: Running, Role: LEADER
I0227 17:24:25.408541 36903 consensus_queue.cc:206] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:25.408679 36758 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:24:25.409204 36758 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:24:25.409516 36903 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd34c4935164b99889130f50a1ac965. Latest consensus state: current_term: 2 leader_uuid: "bbd34c4935164b99889130f50a1ac965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:25.409541 36903 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:25.412792 36758 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:24:25.412884 36758 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/data to 2721 blocks
I0227 17:24:25.413627 36904 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:24:25.413795 36904 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:25.414638 36758 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I0227 17:24:25.414757 36758 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
uuid: "a9929d795dd545d6ad6ba5918488ad2f"
format_stamp: "Formatted at 2019-02-28 01:24:25 on ve0518.halxg.cloudera.com"
I0227 17:24:25.414878 36758 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:24:25.415354 36904 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:25.415652 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:25.415737 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:25.416465 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:25.416537 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:25.417850 36914 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:24:25.418375 35095 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:25.419612 36914 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:24:25.424427 36904 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I0227 17:24:25.426095 36914 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:24:25.426842 36914 catalog_manager.cc:4106] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965: Loaded TSK: 0
I0227 17:24:25.427826 36914 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:24:25.451867 35126 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Rejecting Update request from peer eaae685ab2234943bccdfa4e9d6b87e1 for earlier term 1. Current term is 2. Ops: []
I0227 17:24:25.453303 37034 consensus_queue.cc:995] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I0227 17:24:25.453496 37034 raft_consensus.cc:2881] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 LEADER]: Stepping down as leader of term 1
I0227 17:24:25.453585 37034 raft_consensus.cc:705] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaae685ab2234943bccdfa4e9d6b87e1, State: Running, Role: LEADER
I0227 17:24:25.453894 37034 consensus_queue.cc:229] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:25.453982 37034 consensus_peers.cc:482] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 -> Peer bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272): Closing peer: bbd34c4935164b99889130f50a1ac965
I0227 17:24:25.454063 37034 consensus_peers.cc:482] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 -> Peer 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035): Closing peer: 45d749848fc844e1af0e5b15b810994f
I0227 17:24:25.454341 37034 raft_consensus.cc:2886] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:24:25.497522 36758 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:24:25.502734 36758 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:24:25.502962 36758 tablet_server_main.cc:90] Starting tablet server...
I0227 17:24:25.532219 36758 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.195:51972
I0227 17:24:25.532709 36758 webserver.cc:175] Starting webserver on 127.112.245.195:0
I0227 17:24:25.532759 36758 webserver.cc:186] Document root disabled
I0227 17:24:25.533454 36758 webserver.cc:313] Webserver started. Bound to: http://127.112.245.195:46551/
I0227 17:24:25.534090 36758 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
I0227 17:24:25.538988 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 36758
I0227 17:24:25.541460 36758 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:24:25.567466 37278 heartbeater.cc:345] Connected to a master server at 127.112.245.253:41272
I0227 17:24:25.567596 37278 heartbeater.cc:425] Registering TS with master...
I0227 17:24:25.567982 37278 heartbeater.cc:474] Master 127.112.245.253:41272 requested a full tablet report, sending...
I0227 17:24:25.568852 35537 ts_manager.cc:98] Registered new tserver with Master: a9929d795dd545d6ad6ba5918488ad2f (127.112.245.195:51972)
I0227 17:24:25.570622 35537 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.195:51370
I0227 17:24:25.570652 37276 heartbeater.cc:345] Connected to a master server at 127.112.245.252:34480
I0227 17:24:25.570696 37276 heartbeater.cc:425] Registering TS with master...
I0227 17:24:25.570837 37276 heartbeater.cc:474] Master 127.112.245.252:34480 requested a full tablet report, sending...
I0227 17:24:25.571579 36091 ts_manager.cc:98] Registered new tserver with Master: a9929d795dd545d6ad6ba5918488ad2f (127.112.245.195:51972)
I0227 17:24:25.580404 37279 heartbeater.cc:345] Connected to a master server at 127.112.245.254:39035
I0227 17:24:25.580466 37279 heartbeater.cc:425] Registering TS with master...
I0227 17:24:25.580657 37279 heartbeater.cc:474] Master 127.112.245.254:39035 requested a full tablet report, sending...
I0227 17:24:25.581359 35117 ts_manager.cc:98] Registered new tserver with Master: a9929d795dd545d6ad6ba5918488ad2f (127.112.245.195:51972)
I0227 17:24:25.588266 115671 external_mini_cluster.cc:623] 3 TS(s) registered with all masters
I0227 17:24:25.605850 36197 catalog_manager.cc:1130] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:24:25.614015 36196 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: acquiring CA information for follower catalog manager: success
I0227 17:24:25.615330 36196 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:24:25.652611 36745 heartbeater.cc:466] Master 127.112.245.252:34480 was elected leader, sending a full tablet report...
I0227 17:24:25.863317 36407 heartbeater.cc:466] Master 127.112.245.252:34480 was elected leader, sending a full tablet report...
I0227 17:24:25.938491 36096 raft_consensus.cc:1181] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 2 FOLLOWER]: Refusing update from remote peer bbd34c4935164b99889130f50a1ac965: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I0227 17:24:25.939419 37567 consensus_queue.cc:984] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I0227 17:24:25.940785 37035 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd34c4935164b99889130f50a1ac965. Latest consensus state: current_term: 2 leader_uuid: "bbd34c4935164b99889130f50a1ac965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:25.940822 37035 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:25.952409 37567 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bbd34c4935164b99889130f50a1ac965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:25.952452 37567 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:25.953330 35126 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Refusing update from remote peer bbd34c4935164b99889130f50a1ac965: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I0227 17:24:25.953817 37567 consensus_queue.cc:984] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I0227 17:24:25.953899 37035 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bbd34c4935164b99889130f50a1ac965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:25.953930 37035 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:25.959530 37605 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd34c4935164b99889130f50a1ac965. Latest consensus state: current_term: 2 leader_uuid: "bbd34c4935164b99889130f50a1ac965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:25.959571 37605 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:25.959686 37605 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bbd34c4935164b99889130f50a1ac965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:25.959712 37605 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:25.971742 35537 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.253:34721:
name: "default.test_delete_table_sync"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:24:25.972930 35537 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.test_delete_table_sync in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:24:26.072459 37278 heartbeater.cc:466] Master 127.112.245.253:41272 was elected leader, sending a full tablet report...
I0227 17:24:26.149296 36746 heartbeater.cc:466] Master 127.112.245.253:41272 was elected leader, sending a full tablet report...
I0227 17:24:26.368415 36408 heartbeater.cc:466] Master 127.112.245.253:41272 was elected leader, sending a full tablet report...
I0227 17:24:26.457064 37609 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bbd34c4935164b99889130f50a1ac965)
I0227 17:24:26.457204 37609 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:26.457754 37609 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:26.458165 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:26.458241 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:26.458264 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:26.458362 36096 raft_consensus.cc:2346] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d749848fc844e1af0e5b15b810994f in term 2.
I0227 17:24:26.468438 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1; no voters: bbd34c4935164b99889130f50a1ac965
I0227 17:24:26.468739 37609 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Leader pre-election won for term 3
I0227 17:24:26.468763 37609 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Starting leader election (detected failure of leader bbd34c4935164b99889130f50a1ac965)
I0227 17:24:26.468781 37609 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 2 FOLLOWER]: Advancing to term 3
I0227 17:24:26.469990 37609 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:26.470435 37609 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 3 election: Requested vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:26.470783 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965"
I0227 17:24:26.470854 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 LEADER]: Leader election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:26.470860 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1"
I0227 17:24:26.470921 36096 raft_consensus.cc:2886] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 2 FOLLOWER]: Advancing to term 3
I0227 17:24:26.472097 36096 raft_consensus.cc:2346] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d749848fc844e1af0e5b15b810994f in term 3.
I0227 17:24:26.474874 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1; no voters: bbd34c4935164b99889130f50a1ac965
I0227 17:24:26.475365 37609 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 FOLLOWER]: Leader election won for term 3
I0227 17:24:26.475527 37609 raft_consensus.cc:667] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 LEADER]: Becoming Leader. State: Replica: 45d749848fc844e1af0e5b15b810994f, State: Running, Role: LEADER
I0227 17:24:26.475945 37609 consensus_queue.cc:206] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:26.476944 37609 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45d749848fc844e1af0e5b15b810994f. Latest consensus state: current_term: 3 leader_uuid: "45d749848fc844e1af0e5b15b810994f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:26.476967 37609 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: LEADER
I0227 17:24:26.482378 38367 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:24:26.485029 38367 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:24:26.491009 38367 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:24:26.491538 38367 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: Loaded TSK: 0
I0227 17:24:26.492255 38367 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:24:26.519471 38378 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bbd34c4935164b99889130f50a1ac965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:26.519511 38378 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:26.519645 35126 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 LEADER]: Rejecting Update request from peer bbd34c4935164b99889130f50a1ac965 for earlier term 2. Current term is 3. Ops: []
I0227 17:24:26.520095 38378 consensus_queue.cc:995] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I0227 17:24:26.520252 38378 consensus_queue.cc:1162] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [LEADER]: Peer 45d749848fc844e1af0e5b15b810994f log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I0227 17:24:26.520368 38378 raft_consensus.cc:2881] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 LEADER]: Stepping down as leader of term 2
I0227 17:24:26.520449 38378 raft_consensus.cc:705] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbd34c4935164b99889130f50a1ac965, State: Running, Role: LEADER
I0227 17:24:26.524591 38378 consensus_queue.cc:229] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:26.524706 38378 consensus_peers.cc:482] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 -> Peer eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480): Closing peer: eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:26.524802 38378 consensus_peers.cc:482] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 -> Peer 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035): Closing peer: 45d749848fc844e1af0e5b15b810994f
I0227 17:24:26.525022 38378 raft_consensus.cc:2886] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 2 FOLLOWER]: Advancing to term 3
W0227 17:24:26.526690 38381 catalog_manager.cc:980] Failed waiting for node to catch up after master election: Illegal state: Node bbd34c4935164b99889130f50a1ac965 not leader. Raft Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:26.725003 36223 catalog_manager.cc:1130] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:24:26.981583 38725 raft_consensus.cc:472] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:24:26.981717 38725 raft_consensus.cc:494] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:26.982273 38725 leader_election.cc:292] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272)
I0227 17:24:26.982688 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:26.982744 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:26.982780 35555 raft_consensus.cc:2346] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaae685ab2234943bccdfa4e9d6b87e1 in term 3.
I0227 17:24:26.982847 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate eaae685ab2234943bccdfa4e9d6b87e1 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:26.983093 35936 leader_election.cc:306] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1; no voters: 
I0227 17:24:26.983587 38725 raft_consensus.cc:2640] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I0227 17:24:26.983616 38725 raft_consensus.cc:472] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:24:26.983640 38725 raft_consensus.cc:2886] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 3 FOLLOWER]: Advancing to term 4
I0227 17:24:26.987279 38725 raft_consensus.cc:494] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:26.987741 38725 leader_election.cc:292] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 4 election: Requested vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272)
I0227 17:24:26.988107 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965"
I0227 17:24:26.988123 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f"
I0227 17:24:26.988173 35555 raft_consensus.cc:2886] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 3 FOLLOWER]: Advancing to term 4
I0227 17:24:26.988200 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 LEADER]: Leader election vote request: Denying vote to candidate eaae685ab2234943bccdfa4e9d6b87e1 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:26.989446 35555 raft_consensus.cc:2346] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaae685ab2234943bccdfa4e9d6b87e1 in term 4.
I0227 17:24:26.989792 35936 leader_election.cc:306] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1; no voters: 45d749848fc844e1af0e5b15b810994f
I0227 17:24:26.990130 38725 raft_consensus.cc:2640] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 FOLLOWER]: Leader election won for term 4
I0227 17:24:26.990365 38725 raft_consensus.cc:667] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Becoming Leader. State: Replica: eaae685ab2234943bccdfa4e9d6b87e1, State: Running, Role: LEADER
I0227 17:24:26.990661 38725 consensus_queue.cc:206] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:26.991443 38725 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaae685ab2234943bccdfa4e9d6b87e1. Latest consensus state: current_term: 4 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:26.991536 38725 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:26.993285 38729 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:24:26.994707 38729 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:24:26.995993 38729 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:24:26.996651 38729 catalog_manager.cc:4106] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1: Loaded TSK: 0
I0227 17:24:26.997190 38729 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:24:27.010201 35555 raft_consensus.cc:1149] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Rejecting Update request from peer 45d749848fc844e1af0e5b15b810994f for earlier term 3. Current term is 4. Ops: []
I0227 17:24:27.011535 38733 consensus_queue.cc:995] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I0227 17:24:27.011893 38733 raft_consensus.cc:2881] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 LEADER]: Stepping down as leader of term 3
I0227 17:24:27.011986 38733 raft_consensus.cc:705] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 45d749848fc844e1af0e5b15b810994f, State: Running, Role: LEADER
I0227 17:24:27.012281 38733 consensus_queue.cc:229] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:27.012406 38733 consensus_peers.cc:482] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f -> Peer eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480): Closing peer: eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:27.012500 38733 consensus_peers.cc:482] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f -> Peer bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272): Closing peer: bbd34c4935164b99889130f50a1ac965
I0227 17:24:27.012706 38733 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 3 FOLLOWER]: Advancing to term 4
I0227 17:24:27.086520 37279 heartbeater.cc:466] Master 127.112.245.254:39035 was elected leader, sending a full tablet report...
I0227 17:24:27.148840 36747 heartbeater.cc:466] Master 127.112.245.254:39035 was elected leader, sending a full tablet report...
W0227 17:24:27.178737 36221 hms_client.cc:245] Time spent create HMS table: real 1.204s	user 0.001s	sys 0.000s
W0227 17:24:27.179913 35537 catalog_manager.cc:1636] Illegal state: an error occurred while writing to the sys-catalog: Replica bbd34c4935164b99889130f50a1ac965 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:27.367748 36409 heartbeater.cc:466] Master 127.112.245.254:39035 was elected leader, sending a full tablet report...
I0227 17:24:27.498682 35126 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Refusing update from remote peer eaae685ab2234943bccdfa4e9d6b87e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I0227 17:24:27.498728 35126 pending_rounds.cc:78] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: Aborting all transactions after (but not including) 4
I0227 17:24:27.498752 35126 pending_rounds.cc:99] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f: Aborting uncommitted NO_OP operation due to leader change: 3.5
I0227 17:24:27.498874 35126 raft_consensus.cc:2723] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I0227 17:24:27.499507 38727 consensus_queue.cc:984] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I0227 17:24:27.500567 38737 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaae685ab2234943bccdfa4e9d6b87e1. Latest consensus state: current_term: 4 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:27.500599 38737 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:27.510128 38727 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:27.510172 38727 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:27.510578 38737 sys_catalog.cc:337] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:27.510610 38737 sys_catalog.cc:340] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:27.510627 35555 raft_consensus.cc:1181] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Refusing update from remote peer eaae685ab2234943bccdfa4e9d6b87e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I0227 17:24:27.511097 38727 consensus_queue.cc:984] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I0227 17:24:27.523696 39141 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaae685ab2234943bccdfa4e9d6b87e1. Latest consensus state: current_term: 4 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:27.523741 39141 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:27.523850 39141 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:27.523878 39141 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:24:27.578155 37276 heartbeater.cc:466] Master 127.112.245.252:34480 was elected leader, sending a full tablet report...
I0227 17:24:27.662397 36745 heartbeater.cc:466] Master 127.112.245.252:34480 was elected leader, sending a full tablet report...
I0227 17:24:27.869592 36407 heartbeater.cc:466] Master 127.112.245.252:34480 was elected leader, sending a full tablet report...
I0227 17:24:27.903757 39134 sys_catalog.cc:337] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } } }
I0227 17:24:27.903795 39134 sys_catalog.cc:340] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [sys.catalog]: This master's current role is: LEADER
I0227 17:24:28.446074 39617 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:28.446207 39617 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:28.446750 39617 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:28.447165 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:28.447243 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:28.447259 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:28.447332 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:28.447736 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:28.449156 39617 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:28.807312 39822 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:28.807476 39822 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:28.808027 39822 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:28.808542 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:28.808542 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:28.808637 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:28.808638 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:28.809034 35507 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:28.810087 39822 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:29.323139 40024 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:29.330207 40024 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:29.330950 40024 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:29.331228 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:29.331300 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:29.331315 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:29.331390 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:29.331758 35507 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:29.331929 40024 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:29.479966 40169 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:29.480160 40169 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:29.481132 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:29.481160 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:29.481211 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:29.481240 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:29.481336 40169 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:29.481561 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:29.481786 40169 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:30.099756 40374 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:30.099889 40374 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:30.100464 40374 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:30.100872 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:30.100878 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:30.100953 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:30.100955 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:30.101297 35095 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:30.102733 40374 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:30.410137 40607 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:30.410260 40607 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:30.410800 40607 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:30.411265 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:30.411361 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:30.412808 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:30.412897 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:30.413345 35509 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:30.418074 40607 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:31.149957 41313 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:31.151048 41313 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:31.153466 41313 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:31.153501 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:31.153570 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:31.153576 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:31.153656 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:31.154155 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:31.154542 41313 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:31.685724 41652 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:31.686007 41652 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:31.686697 41652 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:31.687137 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:31.687177 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:31.687228 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:31.687264 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:31.687628 35095 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:31.687762 41652 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:31.768003 41656 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:31.768213 41656 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:31.772404 41656 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:31.772879 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:31.772909 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:31.772970 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:31.773008 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:31.773458 35507 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:31.773628 41656 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:32.306176 41923 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:32.306414 41923 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:32.307605 41923 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:32.307916 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:32.307955 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:32.308013 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:32.308053 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:32.308467 35509 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:32.308750 41923 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
27140 T38 odu.Log4JLogger.warn WARN Metadata has jdbc-type of null yet this is not valid. Ignored
I0227 17:24:32.623575 42275 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:32.623690 42275 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:32.624192 42275 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:32.624622 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:32.624727 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:32.626571 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:32.626659 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:32.627007 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:32.627200 42275 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:32.863721 42492 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:32.863835 42492 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:32.864315 42492 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:32.864912 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:32.865016 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:32.865063 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:32.865156 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:32.865507 35509 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:32.865661 42492 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:33.248150 43135 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:33.248368 43135 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:33.249001 43135 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:33.249442 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:33.249450 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:33.249522 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:33.249531 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:33.250591 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:33.250991 43135 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:33.424360 43438 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:33.424567 43438 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:33.425273 43438 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:33.425664 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:33.425730 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:33.425752 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:33.425817 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:33.426179 35509 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:33.426335 43438 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:33.798688 43836 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:33.798837 43836 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:33.799423 43836 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:33.799836 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:33.799852 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:33.799907 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:33.799923 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:33.800278 35095 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:33.800405 43836 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:34.767035 44767 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:34.767393 44767 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:34.768193 44767 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:34.768476 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:34.768488 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:34.768569 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:34.768571 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:34.768939 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:34.769075 44767 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:34.951721 44952 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:34.951921 44952 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:34.952802 44952 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:34.953246 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:34.953295 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:34.953335 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:34.953372 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:34.953706 35507 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:34.953848 44952 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:35.333079 45323 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:35.333567 45323 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:35.334213 45323 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:35.334637 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:35.334643 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:35.334723 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:35.334736 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:35.335083 35095 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:35.335249 45323 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:35.538436 45488 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:35.538614 45488 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:35.539367 45488 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:35.539582 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:35.539674 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:35.539849 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:35.539934 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:35.540333 35509 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:35.540488 45488 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W0227 17:24:35.617161 37353 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.253:41272)
I0227 17:24:35.871323 45816 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:35.871565 45816 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:35.872490 45816 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:35.872792 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:35.872881 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:35.872913 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:35.873009 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:35.873353 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:35.873478 45816 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:35.908306 36091 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.252:47114:
name: "default.test_delete_table_sync"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:24:35.909664 36091 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.test_delete_table_sync in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:24:36.398603 46359 raft_consensus.cc:472] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:36.398736 46359 raft_consensus.cc:494] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:36.399276 46359 leader_election.cc:292] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 45d749848fc844e1af0e5b15b810994f (127.112.245.254:39035), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:36.399605 35126 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "45d749848fc844e1af0e5b15b810994f" is_pre_election: true
I0227 17:24:36.399696 35126 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:36.399765 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbd34c4935164b99889130f50a1ac965" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:36.399850 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate bbd34c4935164b99889130f50a1ac965 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:36.400197 35509 leader_election.cc:306] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd34c4935164b99889130f50a1ac965; no voters: 45d749848fc844e1af0e5b15b810994f, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:36.403626 46359 raft_consensus.cc:2588] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:24:36.879338 46673 raft_consensus.cc:472] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eaae685ab2234943bccdfa4e9d6b87e1)
I0227 17:24:36.879508 46673 raft_consensus.cc:494] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d749848fc844e1af0e5b15b810994f" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 39035 } } peers { permanent_uuid: "bbd34c4935164b99889130f50a1ac965" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 41272 } } peers { permanent_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 34480 } }
I0227 17:24:36.881295 46673 leader_election.cc:292] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bbd34c4935164b99889130f50a1ac965 (127.112.245.253:41272), eaae685ab2234943bccdfa4e9d6b87e1 (127.112.245.252:34480)
I0227 17:24:36.881652 35555 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bbd34c4935164b99889130f50a1ac965" is_pre_election: true
I0227 17:24:36.881727 35555 raft_consensus.cc:2307] T 00000000000000000000000000000000 P bbd34c4935164b99889130f50a1ac965 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:36.890421 36096 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45d749848fc844e1af0e5b15b810994f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eaae685ab2234943bccdfa4e9d6b87e1" is_pre_election: true
I0227 17:24:36.890499 36096 raft_consensus.cc:2307] T 00000000000000000000000000000000 P eaae685ab2234943bccdfa4e9d6b87e1 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 45d749848fc844e1af0e5b15b810994f for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:24:36.890918 35097 leader_election.cc:306] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d749848fc844e1af0e5b15b810994f; no voters: bbd34c4935164b99889130f50a1ac965, eaae685ab2234943bccdfa4e9d6b87e1
I0227 17:24:36.891137 46673 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 45d749848fc844e1af0e5b15b810994f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
17:24:37.329 [HMS - ERROR - pool-8-thread-2] (RetryingHMSHandler.java:204) AlreadyExistsException(message:Table test_delete_table_sync already exists)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1452)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1539)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
	at com.sun.proxy.$Proxy24.create_table_with_environment_context(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11191)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11175)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

W0227 17:24:37.340603 36192 hms_client.cc:245] Time spent create HMS table: real 1.430s	user 0.001s	sys 0.000s
W0227 17:24:37.341156 36091 catalog_manager.cc:1615] Already present: an error occurred while creating table default.test_delete_table_sync in the HMS: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table test_delete_table_sync already exists)
I0227 17:24:37.341943 36091 rpcz_store.cc:269] Call kudu.master.MasterService.CreateTable from 127.112.245.252:47114 (request call id 3) took 1721ms. Request Metrics: {"Hive Metastore.queue_time_us":58,"Hive Metastore.run_cpu_time_us":914,"Hive Metastore.run_wall_time_us":1430039}
/data/1/adar/apache-kudu-1.9.0/src/kudu/integration-tests/master_failover-itest.cc:238: Failure
Failed
Bad status: Already present: Error creating table default.test_delete_table_sync on the master: an error occurred while creating table default.test_delete_table_sync in the HMS: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table test_delete_table_sync already exists)
I0227 17:24:37.342872 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 36160
W0227 17:24:37.374658 35936 connection.cc:647] server connection from 127.112.245.193:41840 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:41840): Connection reset by peer (error 104)
W0227 17:24:37.375293 35094 connection.cc:647] server connection from 127.112.245.193:54700 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:54700): Connection reset by peer (error 104)
I0227 17:24:37.375732 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 36413
W0227 17:24:37.375733 35509 connection.cc:647] server connection from 127.112.245.193:43894 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:43894): Connection reset by peer (error 104)
W0227 17:24:37.416990 35507 connection.cc:647] server connection from 127.112.245.194:38015 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:38015): Connection reset by peer (error 104)
W0227 17:24:37.432060 35935 connection.cc:647] server connection from 127.112.245.194:47912 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:47912): Connection reset by peer (error 104)
W0227 17:24:37.432117 35096 connection.cc:647] server connection from 127.112.245.194:60430 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:60430): Connection reset by peer (error 104)
I0227 17:24:37.432440 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 36758
W0227 17:24:37.454584 35095 connection.cc:647] server connection from 127.112.245.195:38177 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:38177): Connection reset by peer (error 104)
W0227 17:24:37.454629 35506 connection.cc:647] server connection from 127.112.245.195:51370 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:51370): Connection reset by peer (error 104)
I0227 17:24:37.454985 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 34769
W0227 17:24:37.527603 35936 connection.cc:647] server connection from 127.112.245.195:51331 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:51331): Connection reset by peer (error 104)
I0227 17:24:37.535342 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 35155
I0227 17:24:37.595713 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 35585
Shutting down hive metastore.
I0227 17:24:39.131947 115671 test_util.cc:135] -----------------------------------------------
I0227 17:24:39.132117 115671 test_util.cc:136] Had fatal failures, leaving test files at /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestDeleteTableSync_1.1551317010817988-115671
[  FAILED  ] HmsConfigurations/MasterFailoverTest.TestDeleteTableSync/1, where GetParam() = 2 (35567 ms)
[ RUN      ] HmsConfigurations/MasterFailoverTest.TestKUDU1374/1
2019-02-27 17:24:41: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1
log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hive-498021fa15186aee8b282d3c032fbd2cede6bec4/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
5846 T1 oahu.NativeCodeLoader.<clinit> WARN Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting hive metastore on port 0
7107 T1 odu.Log4JLogger.info Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
7109 T1 odu.Log4JLogger.info Property datanucleus.cache.level2 unknown - will be ignored
W0227 17:25:12.610230 115671 mini_hms.cc:88] Time spent Starting HMS: real 33.477s	user 0.015s	sys 0.148s
I0227 17:25:12.611104 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:59500
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--hive_metastore_uris=thrift://127.0.0.1:47787 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:12.850652 82109 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:12.850958 82109 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:12.852409 82109 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:47787
W0227 17:25:12.852438 82109 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:25:12.852460 82109 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:12.852473 82109 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:25:12.852489 82109 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:25:12.855298 82109 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:47787
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:59500
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.82109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:12.884238 82109 master_main.cc:80] Initializing master server...
I0227 17:25:12.885443 82109 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207275us
I0227 17:25:12.891857 82109 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:12.892014 82109 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:12.892395 82109 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:12.892472 82109 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:12.895628 82109 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data/instance:
uuid: "fb391df2cca34c75b7bbcdb28021f859"
format_stamp: "Formatted at 2019-02-28 01:25:12 on ve0518.halxg.cloudera.com"
I0227 17:25:12.900503 82109 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/wal/instance:
uuid: "fb391df2cca34c75b7bbcdb28021f859"
format_stamp: "Formatted at 2019-02-28 01:25:12 on ve0518.halxg.cloudera.com"
I0227 17:25:12.926728 82109 fs_manager.cc:503] Time spent creating directory manager: real 0.026s	user 0.007s	sys 0.002s
I0227 17:25:12.927716 82109 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:12.928369 82109 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:12.931931 82109 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:12.932049 82109 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data/data to 2721 blocks
I0227 17:25:12.934209 82109 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I0227 17:25:12.934312 82109 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/wal
uuid: "fb391df2cca34c75b7bbcdb28021f859"
format_stamp: "Formatted at 2019-02-28 01:25:12 on ve0518.halxg.cloudera.com"
I0227 17:25:12.934438 82109 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:12.997097 82109 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:13.001868 82109 master_main.cc:83] Starting Master server...
I0227 17:25:13.011617 82109 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.254:59500
I0227 17:25:13.013236 82109 webserver.cc:175] Starting webserver on 127.112.245.254:0
I0227 17:25:13.013260 82109 webserver.cc:186] Document root disabled
I0227 17:25:13.013974 82109 webserver.cc:313] Webserver started. Bound to: http://127.112.245.254:56339/
I0227 17:25:13.014663 82109 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
I0227 17:25:13.016296 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 82109
I0227 17:25:13.016897 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:56207
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--hive_metastore_uris=thrift://127.0.0.1:47787 with env {}
I0227 17:25:13.018671 82339 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:13.028933 82339 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:13.055845 82339 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:25:13.056947 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:56207: Network error: Client connection negotiation failed: client connection to 127.112.245.253:56207: connect: Connection refused (error 111)
I0227 17:25:13.076326 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } attempt: 1
W0227 17:25:13.077142 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:56207: Network error: Client connection negotiation failed: client connection to 127.112.245.253:56207: connect: Connection refused (error 111)
I0227 17:25:13.125283 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } attempt: 2
W0227 17:25:13.126081 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:56207: Network error: Client connection negotiation failed: client connection to 127.112.245.253:56207: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:13.164855 82340 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:13.165277 82340 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:13.169507 82340 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:47787
W0227 17:25:13.169587 82340 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:25:13.169653 82340 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:13.169705 82340 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:25:13.169777 82340 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:25:13.172704 82340 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:47787
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:56207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.82340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:13.183547 82340 master_main.cc:80] Initializing master server...
I0227 17:25:13.184608 82340 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207775us
I0227 17:25:13.186812 82340 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:13.186944 82340 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:13.187376 82340 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:13.187474 82340 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:13.190671 82340 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data/instance:
uuid: "085345f98902407db9fdb0edaf1c1438"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:13.191362 82340 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/wal/instance:
uuid: "085345f98902407db9fdb0edaf1c1438"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:13.218426 82340 fs_manager.cc:503] Time spent creating directory manager: real 0.027s	user 0.006s	sys 0.002s
I0227 17:25:13.219414 82340 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:13.220022 82340 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:13.224486 82340 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:13.224675 82340 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data/data to 2721 blocks
I0227 17:25:13.227907 82340 fs_manager.cc:419] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I0227 17:25:13.228168 82340 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/wal
uuid: "085345f98902407db9fdb0edaf1c1438"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:13.228371 82340 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:13.237195 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } attempt: 3
W0227 17:25:13.237962 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:56207: Network error: Client connection negotiation failed: client connection to 127.112.245.253:56207: connect: Connection refused (error 111)
I0227 17:25:13.306556 82340 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:13.307636 82340 master_main.cc:83] Starting Master server...
I0227 17:25:13.316584 82340 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.253:56207
I0227 17:25:13.317071 82340 webserver.cc:175] Starting webserver on 127.112.245.253:0
I0227 17:25:13.317088 82340 webserver.cc:186] Document root disabled
I0227 17:25:13.317795 82340 webserver.cc:313] Webserver started. Bound to: http://127.112.245.253:52720/
I0227 17:25:13.318451 82340 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
I0227 17:25:13.322932 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 82340
I0227 17:25:13.323576 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:54641
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--hive_metastore_uris=thrift://127.0.0.1:47787 with env {}
I0227 17:25:13.331427 82609 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:13.336410 82609 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:13.346985 82609 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:13.354861 82609 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:25:13.355468 82609 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
I0227 17:25:13.375617 82609 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 1
W0227 17:25:13.377455 82609 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
I0227 17:25:13.414633 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } attempt: 4
I0227 17:25:13.419456 82339 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:25:13.420418 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
I0227 17:25:13.425595 82609 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 2
W0227 17:25:13.426280 82609 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
I0227 17:25:13.483590 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 1
W0227 17:25:13.484480 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:13.528391 82614 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:13.528714 82614 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:13.530047 82614 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:47787
W0227 17:25:13.530086 82614 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:25:13.530117 82614 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:13.530138 82614 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:25:13.530170 82614 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:25:13.533099 82614 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:47787
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:54641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.82614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:13.538101 82609 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 3
I0227 17:25:13.538633 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 2
W0227 17:25:13.538869 82609 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
W0227 17:25:13.539541 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
I0227 17:25:13.544101 82614 master_main.cc:80] Initializing master server...
I0227 17:25:13.545258 82614 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207775us
I0227 17:25:13.546953 82614 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:13.547158 82614 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:13.547508 82614 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:13.548223 82614 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:13.552179 82614 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data/instance:
uuid: "5f2313af35794c1ab241048f12ffc6a4"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:13.553125 82614 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/wal/instance:
uuid: "5f2313af35794c1ab241048f12ffc6a4"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:13.571099 82614 fs_manager.cc:503] Time spent creating directory manager: real 0.016s	user 0.007s	sys 0.001s
I0227 17:25:13.571941 82614 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:13.572517 82614 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:13.575161 82614 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:13.575233 82614 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:25:13.577823 82614 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.001s
I0227 17:25:13.577911 82614 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "5f2313af35794c1ab241048f12ffc6a4"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:13.577991 82614 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:13.619686 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 3
W0227 17:25:13.620656 82339 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
I0227 17:25:13.714993 82609 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 4
W0227 17:25:13.715703 82609 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:54641: Network error: Client connection negotiation failed: client connection to 127.112.245.252:54641: connect: Connection refused (error 111)
I0227 17:25:13.728063 82614 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:13.736198 82614 master_main.cc:83] Starting Master server...
I0227 17:25:13.751811 82614 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.252:54641
I0227 17:25:13.755092 82614 webserver.cc:175] Starting webserver on 127.112.245.252:0
I0227 17:25:13.755157 82614 webserver.cc:186] Document root disabled
I0227 17:25:13.756649 82614 webserver.cc:313] Webserver started. Bound to: http://127.112.245.252:40282/
I0227 17:25:13.757639 82614 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
I0227 17:25:13.768302 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 82614
I0227 17:25:13.768978 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.112.245.193:0
--local_ip_for_outbound_sockets=127.112.245.193
--webserver_interface=127.112.245.193
--webserver_port=0
--tserver_master_addrs=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:25:13.777752 83315 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:13.784816 82339 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 4
I0227 17:25:13.794000 83315 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:13.810076 82339 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: Bootstrap starting.
I0227 17:25:13.812142 83315 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:13.813585 82339 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: No blocks or log segments found. Creating new log.
I0227 17:25:13.817965 82339 log.cc:525] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:13.823194 83315 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:13.828776 82339 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: No bootstrap required, opened a new log
I0227 17:25:13.832883 82339 raft_consensus.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:13.832970 82339 raft_consensus.cc:366] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:13.833032 82339 raft_consensus.cc:705] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb391df2cca34c75b7bbcdb28021f859, State: Initialized, Role: FOLLOWER
I0227 17:25:13.833715 82339 consensus_queue.cc:229] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:13.834239 82339 sys_catalog.cc:337] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:13.834311 82339 sys_catalog.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:13.836375 83412 sys_catalog.cc:337] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:13.836403 83412 sys_catalog.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:13.836961 82339 sys_catalog.cc:424] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:25:13.837796 83315 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: Bootstrap starting.
I0227 17:25:13.838296 83414 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0227 17:25:13.838798 82109 master_main.cc:86] Master server successfully started.
I0227 17:25:13.839677 83315 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: No blocks or log segments found. Creating new log.
I0227 17:25:13.840622 83315 log.cc:525] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: Log is configured to *not* fsync() on all Append() calls
W0227 17:25:13.841133 83415 catalog_manager.cc:1104] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:25:13.841168 83415 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:25:13.851514 83315 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: No bootstrap required, opened a new log
I0227 17:25:13.854598 83315 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:13.854691 83315 raft_consensus.cc:366] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:13.854753 83315 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2313af35794c1ab241048f12ffc6a4, State: Initialized, Role: FOLLOWER
I0227 17:25:13.855641 83315 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:13.855976 83315 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:13.856006 83315 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:13.856966 83315 sys_catalog.cc:424] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:25:13.858294 82614 master_main.cc:86] Master server successfully started.
W0227 17:25:13.866232 83442 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:25:13.866334 83442 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:25:13.869488 83441 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 5f2313af35794c1ab241048f12ffc6a4, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:13.878648 83439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:13.878688 83439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: FOLLOWER
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:13.933073 83319 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:13.933351 83319 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:13.934417 83319 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:13.935936 83319 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.193
I0227 17:25:13.942193 83319 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.112.245.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.83319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.193
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:13.964838 83319 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:25:13.969210 83319 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207775us
I0227 17:25:13.970149 83319 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:13.978085 83319 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:13.978552 83319 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:13.978631 83319 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:13.981580 83319 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data/instance:
uuid: "88ecc5dd132749c8ac37c3e22f2f8205"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:13.982044 83319 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/wal/instance:
uuid: "88ecc5dd132749c8ac37c3e22f2f8205"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
W0227 17:25:13.987524 83258 tablet.cc:1786] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:25:14.001374 83319 fs_manager.cc:503] Time spent creating directory manager: real 0.019s	user 0.004s	sys 0.004s
W0227 17:25:14.003387 82292 tablet.cc:1786] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:25:14.005398 83319 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:14.006863 83319 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:14.014119 83319 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:14.014415 83319 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data/data to 2721 blocks
I0227 17:25:14.018833 82609 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } attempt: 5
I0227 17:25:14.022874 83319 fs_manager.cc:419] Time spent opening block manager: real 0.009s	user 0.000s	sys 0.001s
I0227 17:25:14.023010 83319 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/wal
uuid: "88ecc5dd132749c8ac37c3e22f2f8205"
format_stamp: "Formatted at 2019-02-28 01:25:13 on ve0518.halxg.cloudera.com"
I0227 17:25:14.023119 83319 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:14.025274 82609 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: Bootstrap starting.
I0227 17:25:14.028513 82609 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: No blocks or log segments found. Creating new log.
I0227 17:25:14.029793 82609 log.cc:525] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:14.039973 82609 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: No bootstrap required, opened a new log
I0227 17:25:14.043793 82609 raft_consensus.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:14.043876 82609 raft_consensus.cc:366] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:14.043937 82609 raft_consensus.cc:705] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085345f98902407db9fdb0edaf1c1438, State: Initialized, Role: FOLLOWER
I0227 17:25:14.046751 82609 consensus_queue.cc:229] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:14.047266 82609 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.047379 82609 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:14.047310 83860 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.047561 83860 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:14.048679 82609 sys_catalog.cc:424] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:25:14.049975 82340 master_main.cc:86] Master server successfully started.
I0227 17:25:14.057413 83864 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 085345f98902407db9fdb0edaf1c1438, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
W0227 17:25:14.062045 83865 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:25:14.062086 83865 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:25:14.064404 83319 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
W0227 17:25:14.072051 82564 tablet.cc:1786] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:25:14.077289 83319 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:25:14.077446 83319 tablet_server_main.cc:90] Starting tablet server...
I0227 17:25:14.099150 83319 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.193:58465
I0227 17:25:14.099555 83319 webserver.cc:175] Starting webserver on 127.112.245.193:0
I0227 17:25:14.099573 83319 webserver.cc:186] Document root disabled
I0227 17:25:14.100248 83319 webserver.cc:313] Webserver started. Bound to: http://127.112.245.193:48504/
I0227 17:25:14.100754 83319 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
I0227 17:25:14.102133 83319 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:25:14.106956 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 83319
I0227 17:25:14.107635 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data
--block_manager=log
--rpc_bind_addresses=127.112.245.194:0
--local_ip_for_outbound_sockets=127.112.245.194
--webserver_interface=127.112.245.194
--webserver_port=0
--tserver_master_addrs=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:25:14.119262 84000 heartbeater.cc:345] Connected to a master server at 127.112.245.253:56207
I0227 17:25:14.119292 83998 heartbeater.cc:345] Connected to a master server at 127.112.245.252:54641
I0227 17:25:14.119393 84000 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.119426 83998 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.119808 83998 heartbeater.cc:474] Master 127.112.245.252:54641 requested a full tablet report, sending...
I0227 17:25:14.119819 84000 heartbeater.cc:474] Master 127.112.245.253:56207 requested a full tablet report, sending...
I0227 17:25:14.120923 84001 heartbeater.cc:345] Connected to a master server at 127.112.245.254:59500
I0227 17:25:14.120967 84001 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.121117 84001 heartbeater.cc:474] Master 127.112.245.254:59500 requested a full tablet report, sending...
I0227 17:25:14.121522 82575 ts_manager.cc:98] Registered new tserver with Master: 88ecc5dd132749c8ac37c3e22f2f8205 (127.112.245.193:58465)
I0227 17:25:14.121752 83271 ts_manager.cc:98] Registered new tserver with Master: 88ecc5dd132749c8ac37c3e22f2f8205 (127.112.245.193:58465)
I0227 17:25:14.122445 82302 ts_manager.cc:98] Registered new tserver with Master: 88ecc5dd132749c8ac37c3e22f2f8205 (127.112.245.193:58465)
I0227 17:25:14.261001 83439 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:14.261173 83439 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:14.262364 83439 leader_election.cc:292] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500), 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207)
I0227 17:25:14.263157 82578 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f2313af35794c1ab241048f12ffc6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085345f98902407db9fdb0edaf1c1438" is_pre_election: true
I0227 17:25:14.263386 82578 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f2313af35794c1ab241048f12ffc6a4 in term 0.
I0227 17:25:14.263432 82306 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f2313af35794c1ab241048f12ffc6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb391df2cca34c75b7bbcdb28021f859" is_pre_election: true
I0227 17:25:14.263625 82306 raft_consensus.cc:2346] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f2313af35794c1ab241048f12ffc6a4 in term 0.
I0227 17:25:14.263808 82956 leader_election.cc:306] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085345f98902407db9fdb0edaf1c1438, 5f2313af35794c1ab241048f12ffc6a4; no voters: 
I0227 17:25:14.264515 83439 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:25:14.264565 83439 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:14.264595 83439 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:14.265792 83439 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:14.266232 83439 leader_election.cc:292] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [CANDIDATE]: Term 1 election: Requested vote from peers fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500), 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207)
I0227 17:25:14.266499 82306 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f2313af35794c1ab241048f12ffc6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb391df2cca34c75b7bbcdb28021f859"
I0227 17:25:14.266571 82306 raft_consensus.cc:2886] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:14.266624 82578 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f2313af35794c1ab241048f12ffc6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085345f98902407db9fdb0edaf1c1438"
I0227 17:25:14.266687 82578 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:14.267786 82306 raft_consensus.cc:2346] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f2313af35794c1ab241048f12ffc6a4 in term 1.
I0227 17:25:14.267920 82578 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f2313af35794c1ab241048f12ffc6a4 in term 1.
I0227 17:25:14.268080 82959 leader_election.cc:306] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f2313af35794c1ab241048f12ffc6a4, fb391df2cca34c75b7bbcdb28021f859; no voters: 
I0227 17:25:14.268333 83439 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:25:14.268517 83439 raft_consensus.cc:667] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Becoming Leader. State: Replica: 5f2313af35794c1ab241048f12ffc6a4, State: Running, Role: LEADER
I0227 17:25:14.268852 83439 consensus_queue.cc:206] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:14.270961 83439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2313af35794c1ab241048f12ffc6a4. Latest consensus state: current_term: 1 leader_uuid: "5f2313af35794c1ab241048f12ffc6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.271023 83439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:14.271306 84221 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:25:14.272291 84221 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:14.293033 84010 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:14.293314 84010 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:14.294387 84010 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:14.294545 84010 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.194
I0227 17:25:14.296736 84010 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.112.245.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.84010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.194
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:14.306900 84010 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:25:14.309011 84010 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 208275us
I0227 17:25:14.309855 84010 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:14.309955 84010 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:14.310307 84010 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:14.310387 84010 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:14.313279 84010 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data/instance:
uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3"
format_stamp: "Formatted at 2019-02-28 01:25:14 on ve0518.halxg.cloudera.com"
I0227 17:25:14.313755 84010 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/wal/instance:
uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3"
format_stamp: "Formatted at 2019-02-28 01:25:14 on ve0518.halxg.cloudera.com"
I0227 17:25:14.328681 82306 raft_consensus.cc:1181] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Refusing update from remote peer 5f2313af35794c1ab241048f12ffc6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:25:14.328685 82578 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Refusing update from remote peer 5f2313af35794c1ab241048f12ffc6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:25:14.329424 83439 consensus_queue.cc:984] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:14.330042 83439 consensus_queue.cc:984] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:14.332505 84010 fs_manager.cc:503] Time spent creating directory manager: real 0.018s	user 0.005s	sys 0.003s
I0227 17:25:14.333588 83860 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2313af35794c1ab241048f12ffc6a4. Latest consensus state: current_term: 1 leader_uuid: "5f2313af35794c1ab241048f12ffc6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.333631 83860 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:14.334082 83412 sys_catalog.cc:337] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2313af35794c1ab241048f12ffc6a4. Latest consensus state: current_term: 1 leader_uuid: "5f2313af35794c1ab241048f12ffc6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.334120 83412 sys_catalog.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:14.335629 84010 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:14.336156 84010 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:14.338948 83439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f2313af35794c1ab241048f12ffc6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.338987 83439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:14.339190 84010 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:14.342499 84217 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f2313af35794c1ab241048f12ffc6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.342540 84217 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:14.342599 83412 sys_catalog.cc:337] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f2313af35794c1ab241048f12ffc6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.342638 83412 sys_catalog.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:14.344220 84252 mvcc.cc:203] Tried to move safe_time back from 6354194900283387904 to 6354194900047867904. Current Snapshot: MvccSnapshot[committed={T|T < 6354194900283387904}]
I0227 17:25:14.344414 84010 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data/data to 2721 blocks
I0227 17:25:14.344440 83860 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f2313af35794c1ab241048f12ffc6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:14.344471 83860 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:14.346382 84010 fs_manager.cc:419] Time spent opening block manager: real 0.007s	user 0.002s	sys 0.000s
I0227 17:25:14.346566 84010 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/wal
uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3"
format_stamp: "Formatted at 2019-02-28 01:25:14 on ve0518.halxg.cloudera.com"
I0227 17:25:14.346757 84010 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:14.348886 84221 catalog_manager.cc:941] Generated new certificate authority record
I0227 17:25:14.350224 84221 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:25:14.379431 84221 catalog_manager.cc:4095] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: Generated new TSK 0
I0227 17:25:14.380239 84221 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:25:14.389955 84010 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:14.396132 84010 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:25:14.396399 84010 tablet_server_main.cc:90] Starting tablet server...
I0227 17:25:14.436434 84010 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.194:49996
I0227 17:25:14.436899 84010 webserver.cc:175] Starting webserver on 127.112.245.194:0
I0227 17:25:14.436969 84010 webserver.cc:186] Document root disabled
I0227 17:25:14.437683 84010 webserver.cc:313] Webserver started. Bound to: http://127.112.245.194:54750/
I0227 17:25:14.438323 84010 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
I0227 17:25:14.439144 84010 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:25:14.447026 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 84010
I0227 17:25:14.447719 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.112.245.195:0
--local_ip_for_outbound_sockets=127.112.245.195
--webserver_interface=127.112.245.195
--webserver_port=0
--tserver_master_addrs=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:25:14.465567 84508 heartbeater.cc:345] Connected to a master server at 127.112.245.253:56207
I0227 17:25:14.465706 84508 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.466188 84508 heartbeater.cc:474] Master 127.112.245.253:56207 requested a full tablet report, sending...
I0227 17:25:14.466790 84509 heartbeater.cc:345] Connected to a master server at 127.112.245.254:59500
I0227 17:25:14.466930 84509 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.467178 84509 heartbeater.cc:474] Master 127.112.245.254:59500 requested a full tablet report, sending...
I0227 17:25:14.467356 82575 ts_manager.cc:98] Registered new tserver with Master: 3c2b9a85798d4983a0a96b9f73ec0ee3 (127.112.245.194:49996)
I0227 17:25:14.468165 82302 ts_manager.cc:98] Registered new tserver with Master: 3c2b9a85798d4983a0a96b9f73ec0ee3 (127.112.245.194:49996)
I0227 17:25:14.471171 84507 heartbeater.cc:345] Connected to a master server at 127.112.245.252:54641
I0227 17:25:14.471225 84507 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.471458 84507 heartbeater.cc:474] Master 127.112.245.252:54641 requested a full tablet report, sending...
I0227 17:25:14.474622 83271 ts_manager.cc:98] Registered new tserver with Master: 3c2b9a85798d4983a0a96b9f73ec0ee3 (127.112.245.194:49996)
I0227 17:25:14.476336 83271 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.194:42193
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:14.580513 84513 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:14.580865 84513 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:14.581964 84513 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:14.582141 84513 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.195
I0227 17:25:14.584707 84513 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.112.245.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:59500,127.112.245.253:56207,127.112.245.252:54641
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.84513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.195
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:14.595935 84513 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:25:14.598536 84513 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 208275us
I0227 17:25:14.599447 84513 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:14.599561 84513 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:14.599910 84513 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:14.599984 84513 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:14.602870 84513 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data/instance:
uuid: "cbbfafdf0bc146c58b6459f414490a7a"
format_stamp: "Formatted at 2019-02-28 01:25:14 on ve0518.halxg.cloudera.com"
I0227 17:25:14.603736 84513 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/wal/instance:
uuid: "cbbfafdf0bc146c58b6459f414490a7a"
format_stamp: "Formatted at 2019-02-28 01:25:14 on ve0518.halxg.cloudera.com"
I0227 17:25:14.621822 84513 fs_manager.cc:503] Time spent creating directory manager: real 0.018s	user 0.005s	sys 0.002s
I0227 17:25:14.622764 84513 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:14.623330 84513 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:14.624649 83271 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.193:57349
I0227 17:25:14.627037 84513 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:14.627178 84513 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data/data to 2721 blocks
I0227 17:25:14.629134 84513 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I0227 17:25:14.629278 84513 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/wal
uuid: "cbbfafdf0bc146c58b6459f414490a7a"
format_stamp: "Formatted at 2019-02-28 01:25:14 on ve0518.halxg.cloudera.com"
I0227 17:25:14.629434 84513 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:14.692497 84513 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:14.694144 84513 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:25:14.694352 84513 tablet_server_main.cc:90] Starting tablet server...
I0227 17:25:14.719792 84513 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.195:36757
I0227 17:25:14.720191 84513 webserver.cc:175] Starting webserver on 127.112.245.195:0
I0227 17:25:14.720207 84513 webserver.cc:186] Document root disabled
I0227 17:25:14.720860 84513 webserver.cc:313] Webserver started. Bound to: http://127.112.245.195:37047/
I0227 17:25:14.721434 84513 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestKUDU1374_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
I0227 17:25:14.722323 84513 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:25:14.727591 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 84513
I0227 17:25:14.738843 84861 heartbeater.cc:345] Connected to a master server at 127.112.245.252:54641
I0227 17:25:14.739190 84861 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.740762 84861 heartbeater.cc:474] Master 127.112.245.252:54641 requested a full tablet report, sending...
I0227 17:25:14.742343 83271 ts_manager.cc:98] Registered new tserver with Master: cbbfafdf0bc146c58b6459f414490a7a (127.112.245.195:36757)
I0227 17:25:14.742902 84862 heartbeater.cc:345] Connected to a master server at 127.112.245.253:56207
I0227 17:25:14.742954 84862 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.743091 84862 heartbeater.cc:474] Master 127.112.245.253:56207 requested a full tablet report, sending...
I0227 17:25:14.743245 83271 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.195:38262
I0227 17:25:14.743890 82575 ts_manager.cc:98] Registered new tserver with Master: cbbfafdf0bc146c58b6459f414490a7a (127.112.245.195:36757)
I0227 17:25:14.746249 84863 heartbeater.cc:345] Connected to a master server at 127.112.245.254:59500
I0227 17:25:14.746304 84863 heartbeater.cc:425] Registering TS with master...
I0227 17:25:14.746424 84863 heartbeater.cc:474] Master 127.112.245.254:59500 requested a full tablet report, sending...
I0227 17:25:14.747854 82302 ts_manager.cc:98] Registered new tserver with Master: cbbfafdf0bc146c58b6459f414490a7a (127.112.245.195:36757)
I0227 17:25:14.756989 115671 external_mini_cluster.cc:623] 3 TS(s) registered with all masters
I0227 17:25:14.844688 83415 catalog_manager.cc:1102] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: acquiring CA information for follower catalog manager: success
I0227 17:25:14.846032 83415 catalog_manager.cc:1130] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:25:14.880390 84937 raft_consensus.cc:472] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f2313af35794c1ab241048f12ffc6a4)
I0227 17:25:14.881058 84937 raft_consensus.cc:494] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:14.883013 82578 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "085345f98902407db9fdb0edaf1c1438" is_pre_election: true
I0227 17:25:14.883124 82578 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fb391df2cca34c75b7bbcdb28021f859 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:14.883237 84937 leader_election.cc:292] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:14.883384 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4" is_pre_election: true
I0227 17:25:14.883535 83277 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fb391df2cca34c75b7bbcdb28021f859 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:14.884243 82275 leader_election.cc:306] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb391df2cca34c75b7bbcdb28021f859; no voters: 085345f98902407db9fdb0edaf1c1438, 5f2313af35794c1ab241048f12ffc6a4
I0227 17:25:14.884742 84937 raft_consensus.cc:2588] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0227 17:25:14.990244 84507 heartbeater.cc:466] Master 127.112.245.252:54641 was elected leader, sending a full tablet report...
I0227 17:25:15.065382 83865 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: acquiring CA information for follower catalog manager: success
I0227 17:25:15.066762 83865 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:25:15.129736 83998 heartbeater.cc:466] Master 127.112.245.252:54641 was elected leader, sending a full tablet report...
I0227 17:25:15.172364 83271 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.252:39598:
name: "default.test_kudu_1374"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:25:15.173646 83271 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.test_kudu_1374 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:25:15.247216 84861 heartbeater.cc:466] Master 127.112.245.252:54641 was elected leader, sending a full tablet report...
I0227 17:25:15.870209 85494 raft_consensus.cc:472] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f2313af35794c1ab241048f12ffc6a4)
I0227 17:25:15.870349 85494 raft_consensus.cc:494] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:15.870890 85494 leader_election.cc:292] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:15.871276 82578 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "085345f98902407db9fdb0edaf1c1438" is_pre_election: true
I0227 17:25:15.871289 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4" is_pre_election: true
I0227 17:25:15.871351 82578 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fb391df2cca34c75b7bbcdb28021f859 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:15.871358 83277 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fb391df2cca34c75b7bbcdb28021f859 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:15.871695 82274 leader_election.cc:306] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb391df2cca34c75b7bbcdb28021f859; no voters: 085345f98902407db9fdb0edaf1c1438, 5f2313af35794c1ab241048f12ffc6a4
I0227 17:25:15.877920 85494 raft_consensus.cc:2588] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0227 17:25:16.117404 85573 raft_consensus.cc:472] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f2313af35794c1ab241048f12ffc6a4)
I0227 17:25:16.117522 85573 raft_consensus.cc:494] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:16.118803 85573 leader_election.cc:292] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:16.119151 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4" is_pre_election: true
I0227 17:25:16.119156 82306 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fb391df2cca34c75b7bbcdb28021f859" is_pre_election: true
I0227 17:25:16.119225 83277 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 085345f98902407db9fdb0edaf1c1438 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:16.119238 82306 raft_consensus.cc:2307] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 085345f98902407db9fdb0edaf1c1438 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:16.119699 82527 leader_election.cc:306] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 085345f98902407db9fdb0edaf1c1438; no voters: 5f2313af35794c1ab241048f12ffc6a4, fb391df2cca34c75b7bbcdb28021f859
I0227 17:25:16.120137 85573 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W0227 17:25:16.381712 83440 hms_client.cc:245] Time spent create HMS table: real 1.207s	user 0.000s	sys 0.000s
I0227 17:25:16.411314 83271 rpcz_store.cc:269] Call kudu.master.MasterService.CreateTable from 127.112.245.252:39598 (request call id 2) took 1622ms. Request Metrics: {"Hive Metastore.queue_time_us":40,"Hive Metastore.run_cpu_time_us":440,"Hive Metastore.run_wall_time_us":1207188,"child_traces":[["txn",{"apply.queue_time_us":8900,"apply.run_cpu_time_us":471,"apply.run_wall_time_us":1140,"num_ops":2,"prepare.queue_time_us":6865,"prepare.run_cpu_time_us":839,"prepare.run_wall_time_us":1359,"raft.queue_time_us":81,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":433,"replication_time_us":10588,"spinlock_wait_cycles":275584,"thread_start_us":272,"threads_started":3,"wal-append.queue_time_us":10197}]]}
I0227 17:25:16.440017 83924 tablet_service.cc:774] Processing CreateTablet for tablet 4ec4f00807094acead28bf862d918e3d (table=default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:16.440501 83924 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 4ec4f00807094acead28bf862d918e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:16.447891 83924 ts_tablet_manager.cc:1175] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:16.448256 84783 tablet_service.cc:774] Processing CreateTablet for tablet 4ec4f00807094acead28bf862d918e3d (table=default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:16.448325 84427 tablet_service.cc:774] Processing CreateTablet for tablet 4ec4f00807094acead28bf862d918e3d (table=default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:16.448740 84783 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 4ec4f00807094acead28bf862d918e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:16.448797 84427 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 4ec4f00807094acead28bf862d918e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:16.453987 84783 ts_tablet_manager.cc:1175] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:16.457825 85786 ts_tablet_manager.cc:1030] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Bootstrapping tablet
I0227 17:25:16.458108 85786 tablet_bootstrap.cc:439] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Bootstrap starting.
I0227 17:25:16.460155 85795 ts_tablet_manager.cc:1030] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Bootstrapping tablet
I0227 17:25:16.460371 84427 ts_tablet_manager.cc:1175] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:16.460558 85795 tablet_bootstrap.cc:439] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Bootstrap starting.
I0227 17:25:16.465816 85795 tablet_bootstrap.cc:593] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: No blocks or log segments found. Creating new log.
I0227 17:25:16.469348 85799 ts_tablet_manager.cc:1030] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Bootstrapping tablet
I0227 17:25:16.469717 85799 tablet_bootstrap.cc:439] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Bootstrap starting.
I0227 17:25:16.470144 85786 tablet_bootstrap.cc:593] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: No blocks or log segments found. Creating new log.
I0227 17:25:16.470583 85795 log.cc:525] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:16.471236 85786 log.cc:525] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:16.472755 85799 tablet_bootstrap.cc:593] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: No blocks or log segments found. Creating new log.
I0227 17:25:16.473754 85799 log.cc:525] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:16.480585 85795 tablet_bootstrap.cc:439] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: No bootstrap required, opened a new log
I0227 17:25:16.480793 85795 ts_tablet_manager.cc:1047] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.002s
I0227 17:25:16.483928 85795 raft_consensus.cc:340] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.484019 85795 raft_consensus.cc:366] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:16.484064 85795 raft_consensus.cc:705] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbfafdf0bc146c58b6459f414490a7a, State: Initialized, Role: FOLLOWER
I0227 17:25:16.484463 85795 consensus_queue.cc:229] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.485440 85795 ts_tablet_manager.cc:1075] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.001s
I0227 17:25:16.490384 85786 tablet_bootstrap.cc:439] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: No bootstrap required, opened a new log
I0227 17:25:16.490587 85786 ts_tablet_manager.cc:1047] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Time spent bootstrapping tablet: real 0.033s	user 0.002s	sys 0.002s
I0227 17:25:16.493805 85786 raft_consensus.cc:340] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.493901 85786 raft_consensus.cc:366] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:16.493957 85786 raft_consensus.cc:705] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ecc5dd132749c8ac37c3e22f2f8205, State: Initialized, Role: FOLLOWER
I0227 17:25:16.495260 85786 consensus_queue.cc:229] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.495436 85799 tablet_bootstrap.cc:439] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: No bootstrap required, opened a new log
I0227 17:25:16.495635 85799 ts_tablet_manager.cc:1047] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Time spent bootstrapping tablet: real 0.026s	user 0.003s	sys 0.001s
I0227 17:25:16.498803 85786 ts_tablet_manager.cc:1075] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.002s
I0227 17:25:16.498895 85799 raft_consensus.cc:340] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.498991 85799 raft_consensus.cc:366] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:16.499047 85799 raft_consensus.cc:705] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2b9a85798d4983a0a96b9f73ec0ee3, State: Initialized, Role: FOLLOWER
I0227 17:25:16.499634 85799 consensus_queue.cc:229] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.500591 85799 ts_tablet_manager.cc:1075] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.001s
W0227 17:25:16.625499 84002 tablet.cc:1786] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:25:16.660027 85814 raft_consensus.cc:472] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:16.660202 85814 raft_consensus.cc:494] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.661339 85814 leader_election.cc:292] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88ecc5dd132749c8ac37c3e22f2f8205 (127.112.245.193:58465), 3c2b9a85798d4983a0a96b9f73ec0ee3 (127.112.245.194:49996)
I0227 17:25:16.684535 84444 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4ec4f00807094acead28bf862d918e3d" candidate_uuid: "cbbfafdf0bc146c58b6459f414490a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" is_pre_election: true
I0227 17:25:16.684725 84444 raft_consensus.cc:2346] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbbfafdf0bc146c58b6459f414490a7a in term 0.
I0227 17:25:16.684809 83951 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4ec4f00807094acead28bf862d918e3d" candidate_uuid: "cbbfafdf0bc146c58b6459f414490a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" is_pre_election: true
I0227 17:25:16.684964 83951 raft_consensus.cc:2346] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbbfafdf0bc146c58b6459f414490a7a in term 0.
I0227 17:25:16.685420 84730 leader_election.cc:306] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2b9a85798d4983a0a96b9f73ec0ee3, cbbfafdf0bc146c58b6459f414490a7a; no voters: 
I0227 17:25:16.685832 85814 raft_consensus.cc:2640] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:25:16.685884 85814 raft_consensus.cc:472] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:16.685909 85814 raft_consensus.cc:2886] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:16.687150 85814 raft_consensus.cc:494] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.687618 85814 leader_election.cc:292] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [CANDIDATE]: Term 1 election: Requested vote from peers 88ecc5dd132749c8ac37c3e22f2f8205 (127.112.245.193:58465), 3c2b9a85798d4983a0a96b9f73ec0ee3 (127.112.245.194:49996)
I0227 17:25:16.688109 84444 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4ec4f00807094acead28bf862d918e3d" candidate_uuid: "cbbfafdf0bc146c58b6459f414490a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3"
I0227 17:25:16.688190 84444 raft_consensus.cc:2886] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:16.689457 84444 raft_consensus.cc:2346] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbfafdf0bc146c58b6459f414490a7a in term 1.
I0227 17:25:16.689867 84730 leader_election.cc:306] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2b9a85798d4983a0a96b9f73ec0ee3, cbbfafdf0bc146c58b6459f414490a7a; no voters: 
I0227 17:25:16.690500 85814 raft_consensus.cc:2640] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:25:16.690678 85814 raft_consensus.cc:667] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [term 1 LEADER]: Becoming Leader. State: Replica: cbbfafdf0bc146c58b6459f414490a7a, State: Running, Role: LEADER
I0227 17:25:16.690981 85814 consensus_queue.cc:206] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } }
I0227 17:25:16.693044 83951 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4ec4f00807094acead28bf862d918e3d" candidate_uuid: "cbbfafdf0bc146c58b6459f414490a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ecc5dd132749c8ac37c3e22f2f8205"
I0227 17:25:16.693137 83951 raft_consensus.cc:2886] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:16.694350 83951 raft_consensus.cc:2346] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbfafdf0bc146c58b6459f414490a7a in term 1.
I0227 17:25:16.696537 83271 catalog_manager.cc:3791] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a reported cstate change: term changed from 0 to 1, leader changed from <none> to cbbfafdf0bc146c58b6459f414490a7a (127.112.245.195). New cstate: current_term: 1 leader_uuid: "cbbfafdf0bc146c58b6459f414490a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbfafdf0bc146c58b6459f414490a7a" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 36757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 } health_report { overall_health: UNKNOWN } } }
W0227 17:25:16.701292 84510 tablet.cc:1786] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:25:16.723315 84864 tablet.cc:1786] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:25:17.098460 83951 raft_consensus.cc:1181] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205 [term 1 FOLLOWER]: Refusing update from remote peer cbbfafdf0bc146c58b6459f414490a7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:17.099694 86008 consensus_queue.cc:984] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88ecc5dd132749c8ac37c3e22f2f8205" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 58465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:17.113674 84444 raft_consensus.cc:1181] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3 [term 1 FOLLOWER]: Refusing update from remote peer cbbfafdf0bc146c58b6459f414490a7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:17.114768 86008 consensus_queue.cc:984] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c2b9a85798d4983a0a96b9f73ec0ee3" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 49996 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:17.607903 86839 raft_consensus.cc:472] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f2313af35794c1ab241048f12ffc6a4)
I0227 17:25:17.611078 86839 raft_consensus.cc:494] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:17.613868 86839 leader_election.cc:292] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:17.614292 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4" is_pre_election: true
I0227 17:25:17.614367 83277 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 085345f98902407db9fdb0edaf1c1438 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:17.615339 82306 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "fb391df2cca34c75b7bbcdb28021f859" is_pre_election: true
I0227 17:25:17.615406 82306 raft_consensus.cc:2307] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 085345f98902407db9fdb0edaf1c1438 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:17.617442 82527 leader_election.cc:306] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 085345f98902407db9fdb0edaf1c1438; no voters: 5f2313af35794c1ab241048f12ffc6a4, fb391df2cca34c75b7bbcdb28021f859
I0227 17:25:17.619367 86839 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0227 17:25:17.639211 86846 raft_consensus.cc:472] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f2313af35794c1ab241048f12ffc6a4)
I0227 17:25:17.639323 86846 raft_consensus.cc:494] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:17.640079 86846 leader_election.cc:292] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:17.640143 82578 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "085345f98902407db9fdb0edaf1c1438" is_pre_election: true
I0227 17:25:17.640235 82578 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb391df2cca34c75b7bbcdb28021f859 in term 1.
I0227 17:25:17.640547 82274 leader_election.cc:306] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085345f98902407db9fdb0edaf1c1438, fb391df2cca34c75b7bbcdb28021f859; no voters: 
I0227 17:25:17.640681 86846 raft_consensus.cc:2640] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Leader pre-election won for term 2
I0227 17:25:17.640715 86846 raft_consensus.cc:472] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f2313af35794c1ab241048f12ffc6a4)
I0227 17:25:17.640743 86846 raft_consensus.cc:2886] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:25:17.641963 86846 raft_consensus.cc:494] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:17.642318 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4" is_pre_election: true
I0227 17:25:17.642387 83277 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fb391df2cca34c75b7bbcdb28021f859 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:17.642395 86846 leader_election.cc:292] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 election: Requested vote from peers 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:17.642786 82578 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "085345f98902407db9fdb0edaf1c1438"
I0227 17:25:17.642846 82578 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:25:17.644004 82578 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb391df2cca34c75b7bbcdb28021f859 in term 2.
I0227 17:25:17.644268 82274 leader_election.cc:306] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085345f98902407db9fdb0edaf1c1438, fb391df2cca34c75b7bbcdb28021f859; no voters: 
I0227 17:25:17.644557 86846 raft_consensus.cc:2640] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 FOLLOWER]: Leader election won for term 2
I0227 17:25:17.644623 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb391df2cca34c75b7bbcdb28021f859" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4"
I0227 17:25:17.644683 83277 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Leader election vote request: Denying vote to candidate fb391df2cca34c75b7bbcdb28021f859 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:17.645120 86846 raft_consensus.cc:667] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 LEADER]: Becoming Leader. State: Replica: fb391df2cca34c75b7bbcdb28021f859, State: Running, Role: LEADER
I0227 17:25:17.646255 86846 consensus_queue.cc:206] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:17.650133 86846 sys_catalog.cc:337] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb391df2cca34c75b7bbcdb28021f859. Latest consensus state: current_term: 2 leader_uuid: "fb391df2cca34c75b7bbcdb28021f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:17.650246 86846 sys_catalog.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:17.655947 86901 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:25:17.657691 86901 catalog_manager.cc:347] Loaded metadata for table default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]
I0227 17:25:17.661434 86901 catalog_manager.cc:401] Loaded metadata for tablet 4ec4f00807094acead28bf862d918e3d (table default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7])
I0227 17:25:17.661600 86901 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:25:17.662523 86901 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:25:17.665374 86901 catalog_manager.cc:4106] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859: Loaded TSK: 0
I0227 17:25:17.666141 86901 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:25:17.714298 82332 generic_service.cc:148] {username='adar'} at 127.112.245.254:60470 changed flags via RPC: catalog_manager_fail_ts_rpcs from 'false' to 'true'
I0227 17:25:17.721043 82306 raft_consensus.cc:1149] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 LEADER]: Rejecting Update request from peer 5f2313af35794c1ab241048f12ffc6a4 for earlier term 1. Current term is 2. Ops: []
I0227 17:25:17.721546 86976 consensus_queue.cc:995] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I0227 17:25:17.721632 86976 consensus_queue.cc:1162] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [LEADER]: Peer fb391df2cca34c75b7bbcdb28021f859 log is divergent from this leader: its last log entry 2.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I0227 17:25:17.721693 86976 raft_consensus.cc:2881] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Stepping down as leader of term 1
I0227 17:25:17.721711 86976 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5f2313af35794c1ab241048f12ffc6a4, State: Running, Role: LEADER
I0227 17:25:17.721890 86976 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:17.721922 86976 consensus_peers.cc:482] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 -> Peer 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207): Closing peer: 085345f98902407db9fdb0edaf1c1438
I0227 17:25:17.721943 86976 consensus_peers.cc:482] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 -> Peer fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500): Closing peer: fb391df2cca34c75b7bbcdb28021f859
I0227 17:25:17.722093 86976 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 1 FOLLOWER]: Advancing to term 2
W0227 17:25:17.723592 83441 hms_notification_log_listener.cc:130] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events
I0227 17:25:17.724336 82578 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Rejecting Update request from peer 5f2313af35794c1ab241048f12ffc6a4 for earlier term 1. Current term is 2. Ops: []
W0227 17:25:17.726584 84881 master_proxy_rpc.cc:192] Re-attempting AlterTable request to leader Master (127.112.245.252:54641)
I0227 17:25:18.052021 83277 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 2 FOLLOWER]: Refusing update from remote peer fb391df2cca34c75b7bbcdb28021f859: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I0227 17:25:18.053028 86846 consensus_queue.cc:984] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I0227 17:25:18.055111 86983 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb391df2cca34c75b7bbcdb28021f859. Latest consensus state: current_term: 2 leader_uuid: "fb391df2cca34c75b7bbcdb28021f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:18.055146 86983 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:18.056869 82578 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Refusing update from remote peer fb391df2cca34c75b7bbcdb28021f859: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I0227 17:25:18.057320 86877 consensus_queue.cc:984] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I0227 17:25:18.057523 87314 mvcc.cc:203] Tried to move safe_time back from 6354194915535126528 to 6354194913891479552. Current Snapshot: MvccSnapshot[committed={T|T < 6354194915535126528}]
I0227 17:25:18.058007 86877 sys_catalog.cc:337] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fb391df2cca34c75b7bbcdb28021f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:18.058035 86877 sys_catalog.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:18.058645 86983 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fb391df2cca34c75b7bbcdb28021f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:18.058681 86983 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:18.058938 86839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb391df2cca34c75b7bbcdb28021f859. Latest consensus state: current_term: 2 leader_uuid: "fb391df2cca34c75b7bbcdb28021f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:18.058972 86839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:18.059077 86839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fb391df2cca34c75b7bbcdb28021f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:18.059103 86839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:18.060560 82302 catalog_manager.cc:2195] Servicing AlterTable request from {username='adar'} at 127.112.245.254:60472:
table { table_name: "default.test_kudu_1374" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "new_col" type: INT32 is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } } modify_external_catalogs: true
I0227 17:25:18.067323 86877 sys_catalog.cc:337] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fb391df2cca34c75b7bbcdb28021f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:18.067360 86877 sys_catalog.cc:340] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:18.071818 87325 mvcc.cc:203] Tried to move safe_time back from 6354194915535126528 to 6354194913891479552. Current Snapshot: MvccSnapshot[committed={T|T < 6354194915535126528}]
I0227 17:25:18.418699 83442 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:25:18.580209 88143 raft_consensus.cc:472] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fb391df2cca34c75b7bbcdb28021f859)
I0227 17:25:18.580396 88143 raft_consensus.cc:494] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:18.581180 88143 leader_election.cc:292] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:18.581485 82306 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "fb391df2cca34c75b7bbcdb28021f859" is_pre_election: true
I0227 17:25:18.581555 82306 raft_consensus.cc:2307] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 085345f98902407db9fdb0edaf1c1438 for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:18.581647 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4" is_pre_election: true
I0227 17:25:18.581755 83277 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085345f98902407db9fdb0edaf1c1438 in term 2.
I0227 17:25:18.582054 82525 leader_election.cc:306] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 085345f98902407db9fdb0edaf1c1438, 5f2313af35794c1ab241048f12ffc6a4; no voters: fb391df2cca34c75b7bbcdb28021f859
I0227 17:25:18.585891 88143 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Leader pre-election won for term 3
I0227 17:25:18.585927 88143 raft_consensus.cc:472] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Starting leader election (detected failure of leader fb391df2cca34c75b7bbcdb28021f859)
I0227 17:25:18.585961 88143 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 2 FOLLOWER]: Advancing to term 3
I0227 17:25:18.587255 88143 raft_consensus.cc:494] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:18.587857 88143 leader_election.cc:292] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 3 election: Requested vote from peers fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500), 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641)
I0227 17:25:18.588313 83277 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "5f2313af35794c1ab241048f12ffc6a4"
I0227 17:25:18.588387 83277 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 2 FOLLOWER]: Advancing to term 3
I0227 17:25:18.588471 82306 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "085345f98902407db9fdb0edaf1c1438" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "fb391df2cca34c75b7bbcdb28021f859"
I0227 17:25:18.588542 82306 raft_consensus.cc:2307] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 LEADER]: Leader election vote request: Denying vote to candidate 085345f98902407db9fdb0edaf1c1438 for term 3 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:18.589612 83277 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085345f98902407db9fdb0edaf1c1438 in term 3.
I0227 17:25:18.589917 82525 leader_election.cc:306] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 085345f98902407db9fdb0edaf1c1438, 5f2313af35794c1ab241048f12ffc6a4; no voters: fb391df2cca34c75b7bbcdb28021f859
I0227 17:25:18.590075 88143 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 3 FOLLOWER]: Leader election won for term 3
I0227 17:25:18.590227 88143 raft_consensus.cc:667] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [term 3 LEADER]: Becoming Leader. State: Replica: 085345f98902407db9fdb0edaf1c1438, State: Running, Role: LEADER
I0227 17:25:18.590569 88143 consensus_queue.cc:206] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:18.591456 88143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085345f98902407db9fdb0edaf1c1438. Latest consensus state: current_term: 3 leader_uuid: "085345f98902407db9fdb0edaf1c1438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:18.591483 88143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:18.591715 88190 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:25:18.596909 88190 catalog_manager.cc:347] Loaded metadata for table default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]
I0227 17:25:18.598883 88190 catalog_manager.cc:401] Loaded metadata for tablet 4ec4f00807094acead28bf862d918e3d (table default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7])
I0227 17:25:18.599050 88190 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:25:18.599963 83277 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 3 FOLLOWER]: Rejecting Update request from peer fb391df2cca34c75b7bbcdb28021f859 for earlier term 2. Current term is 3. Ops: []
I0227 17:25:18.601804 88170 consensus_queue.cc:995] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 }, Status: INVALID_TERM, Last received: 2.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I0227 17:25:18.602018 88190 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:25:18.602619 88190 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: Loaded TSK: 0
I0227 17:25:18.602444 88212 raft_consensus.cc:2881] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 LEADER]: Stepping down as leader of term 2
I0227 17:25:18.604034 88190 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:25:18.604112 88212 raft_consensus.cc:705] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb391df2cca34c75b7bbcdb28021f859, State: Running, Role: LEADER
I0227 17:25:18.604475 88212 consensus_queue.cc:229] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } }
I0227 17:25:18.604563 88212 consensus_peers.cc:482] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 -> Peer 5f2313af35794c1ab241048f12ffc6a4 (127.112.245.252:54641): Closing peer: 5f2313af35794c1ab241048f12ffc6a4
I0227 17:25:18.604935 88212 consensus_peers.cc:482] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 -> Peer 085345f98902407db9fdb0edaf1c1438 (127.112.245.253:56207): Closing peer: 085345f98902407db9fdb0edaf1c1438
I0227 17:25:18.605140 88212 raft_consensus.cc:2886] T 00000000000000000000000000000000 P fb391df2cca34c75b7bbcdb28021f859 [term 2 FOLLOWER]: Advancing to term 3
I0227 17:25:18.618113 84001 heartbeater.cc:466] Master 127.112.245.254:59500 was elected leader, sending a full tablet report...
I0227 17:25:18.627823 84509 heartbeater.cc:466] Master 127.112.245.254:59500 was elected leader, sending a full tablet report...
I0227 17:25:18.638705 84863 heartbeater.cc:466] Master 127.112.245.254:59500 was elected leader, sending a full tablet report...
W0227 17:25:18.793784 82302 catalog_manager.cc:4003] Failed to send alter table request: Runtime error: Async RPCs configured to fail
I0227 17:25:18.794474 82302 rpcz_store.cc:269] Call kudu.master.MasterService.AlterTable from 127.112.245.254:60472 (request call id 3) took 1065ms. Request Metrics: {"Hive Metastore.queue_time_us":41,"Hive Metastore.run_cpu_time_us":1219,"Hive Metastore.run_wall_time_us":719690,"child_traces":[["txn",{"apply.queue_time_us":40,"apply.run_cpu_time_us":408,"apply.run_wall_time_us":4679,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":617,"prepare.run_wall_time_us":616,"raft.queue_time_us":68,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":235,"replication_time_us":6133,"spinlock_wait_cycles":158592}]]}
I0227 17:25:18.795303 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 82109
W0227 17:25:18.824470 83849 connection.cc:647] client connection to 127.112.245.254:59500 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.254:59500): Connection reset by peer (error 104)
W0227 17:25:18.824501 84384 connection.cc:647] client connection to 127.112.245.254:59500 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.254:59500): Connection reset by peer (error 104)
W0227 17:25:18.824599 84730 connection.cc:647] client connection to 127.112.245.254:59500 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.254:59500): Connection reset by peer (error 104)
W0227 17:25:18.841167 84883 master_proxy_rpc.cc:192] Re-attempting IsAlterTableInProgres request to leader Master (127.112.245.254:59500)
I0227 17:25:19.064712 83277 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [term 3 FOLLOWER]: Refusing update from remote peer 085345f98902407db9fdb0edaf1c1438: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I0227 17:25:19.065299 88172 consensus_queue.cc:984] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I0227 17:25:19.068708 88172 sys_catalog.cc:337] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "085345f98902407db9fdb0edaf1c1438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:19.068742 88172 sys_catalog.cc:340] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 [sys.catalog]: This master's current role is: LEADER
W0227 17:25:19.079064 82527 consensus_peers.cc:457] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438 -> Peer fb391df2cca34c75b7bbcdb28021f859 (127.112.245.254:59500): Couldn't send request to peer fb391df2cca34c75b7bbcdb28021f859. Status: Network error: Client connection negotiation failed: client connection to 127.112.245.254:59500: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I0227 17:25:19.079396 88670 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085345f98902407db9fdb0edaf1c1438. Latest consensus state: current_term: 3 leader_uuid: "085345f98902407db9fdb0edaf1c1438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:19.079401 88663 sys_catalog.cc:337] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "085345f98902407db9fdb0edaf1c1438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391df2cca34c75b7bbcdb28021f859" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 59500 } } peers { permanent_uuid: "085345f98902407db9fdb0edaf1c1438" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 56207 } } peers { permanent_uuid: "5f2313af35794c1ab241048f12ffc6a4" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 54641 } } }
I0227 17:25:19.079464 88663 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:19.079483 88670 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5f2313af35794c1ab241048f12ffc6a4 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:19.119896 84000 heartbeater.cc:466] Master 127.112.245.253:56207 was elected leader, sending a full tablet report...
I0227 17:25:19.121256 82575 catalog_manager.cc:3886] TS 88ecc5dd132749c8ac37c3e22f2f8205 (127.112.245.193:58465) does not have the latest schema for tablet 4ec4f00807094acead28bf862d918e3d (table default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]). Expected version 1 got 0
W0227 17:25:19.130520 84001 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:59500 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.112.245.254:59500: connect: Connection refused (error 111)
W0227 17:25:19.130551 84509 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:59500 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.112.245.254:59500: connect: Connection refused (error 111)
I0227 17:25:19.140535 84862 heartbeater.cc:466] Master 127.112.245.253:56207 was elected leader, sending a full tablet report...
I0227 17:25:19.141986 82575 catalog_manager.cc:3886] TS cbbfafdf0bc146c58b6459f414490a7a (127.112.245.195:36757) does not have the latest schema for tablet 4ec4f00807094acead28bf862d918e3d (table default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]). Expected version 1 got 0
I0227 17:25:19.143113 84508 heartbeater.cc:466] Master 127.112.245.253:56207 was elected leader, sending a full tablet report...
I0227 17:25:19.143945 82575 catalog_manager.cc:3886] TS 3c2b9a85798d4983a0a96b9f73ec0ee3 (127.112.245.194:49996) does not have the latest schema for tablet 4ec4f00807094acead28bf862d918e3d (table default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7]). Expected version 1 got 0
W0227 17:25:19.149929 84863 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:59500 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.112.245.254:59500: connect: Connection refused (error 111)
I0227 17:25:19.161391 88736 tablet.cc:1269] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NOT NULL,
    13:new_col INT32 NULLABLE,
    PRIMARY KEY (key)
) version 1
I0227 17:25:19.164702 88736 tablet.cc:1208] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: MemRowSet was empty: no flush needed.
I0227 17:25:19.169239 88737 tablet.cc:1269] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NOT NULL,
    13:new_col INT32 NULLABLE,
    PRIMARY KEY (key)
) version 1
I0227 17:25:19.169512 88738 tablet.cc:1269] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NOT NULL,
    13:new_col INT32 NULLABLE,
    PRIMARY KEY (key)
) version 1
I0227 17:25:19.169903 88737 tablet.cc:1208] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: MemRowSet was empty: no flush needed.
I0227 17:25:19.170119 88738 tablet.cc:1208] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: MemRowSet was empty: no flush needed.
I0227 17:25:19.182458 88736 tablet.cc:1264] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Skipping requested alter to schema version 1, tablet already version 1
I0227 17:25:19.183416 88738 tablet.cc:1264] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Skipping requested alter to schema version 1, tablet already version 1
I0227 17:25:19.185370 88737 tablet.cc:1264] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Skipping requested alter to schema version 1, tablet already version 1
I0227 17:25:19.186641 88738 tablet.cc:1264] T 4ec4f00807094acead28bf862d918e3d P 88ecc5dd132749c8ac37c3e22f2f8205: Skipping requested alter to schema version 1, tablet already version 1
I0227 17:25:19.187252 88737 tablet.cc:1264] T 4ec4f00807094acead28bf862d918e3d P 3c2b9a85798d4983a0a96b9f73ec0ee3: Skipping requested alter to schema version 1, tablet already version 1
I0227 17:25:19.189131 88736 tablet.cc:1264] T 4ec4f00807094acead28bf862d918e3d P cbbfafdf0bc146c58b6459f414490a7a: Skipping requested alter to schema version 1, tablet already version 1
I0227 17:25:19.192394 82575 catalog_manager.cc:4237] T 00000000000000000000000000000000 P 085345f98902407db9fdb0edaf1c1438: default.test_kudu_1374 [id=590492545c6642e3a9aba7d0131ea9b7] alter complete (version 1)
I0227 17:25:19.367986 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 83319
W0227 17:25:19.383749 82959 connection.cc:647] server connection from 127.112.245.193:57349 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:57349): Connection reset by peer (error 104)
W0227 17:25:19.384357 84728 connection.cc:647] client connection to 127.112.245.193:58465 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:58465): Connection reset by peer (error 104)
W0227 17:25:19.384325 82526 connection.cc:647] server connection from 127.112.245.193:35094 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:35094): Connection reset by peer (error 104)
I0227 17:25:19.384824 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 84010
W0227 17:25:19.403425 82958 connection.cc:647] server connection from 127.112.245.194:42193 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:42193): Connection reset by peer (error 104)
W0227 17:25:19.403528 84730 connection.cc:647] client connection to 127.112.245.194:49996 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:49996): Connection reset by peer (error 104)
I0227 17:25:19.403674 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 84513
W0227 17:25:19.406721 82527 connection.cc:647] server connection from 127.112.245.194:50076 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:50076): Connection reset by peer (error 104)
W0227 17:25:19.423513 82524 connection.cc:647] server connection from 127.112.245.195:48811 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:48811): Connection reset by peer (error 104)
W0227 17:25:19.423560 82956 connection.cc:647] server connection from 127.112.245.195:38262 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:38262): Connection reset by peer (error 104)
I0227 17:25:19.424243 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 82340
I0227 17:25:19.454030 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 82614
Shutting down hive metastore.
[       OK ] HmsConfigurations/MasterFailoverTest.TestKUDU1374/1 (42114 ms)
[ RUN      ] HmsConfigurations/MasterFailoverTest.TestMasterPermanentFailure/1
2019-02-27 17:25:23: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1
log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hive-498021fa15186aee8b282d3c032fbd2cede6bec4/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/1/adar/apache-kudu-1.9.0/thirdparty/src/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
3631 T1 oahu.NativeCodeLoader.<clinit> WARN Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting hive metastore on port 0
4471 T1 odu.Log4JLogger.info Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
4472 T1 odu.Log4JLogger.info Property datanucleus.cache.level2 unknown - will be ignored
I0227 17:25:40.349592 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:35995
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--hive_metastore_uris=thrift://127.0.0.1:54702 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:40.450547 123111 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:40.450984 123111 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:40.452224 123111 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:54702
W0227 17:25:40.452322 123111 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:25:40.452352 123111 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:40.452404 123111 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:25:40.452426 123111 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:25:40.454874 123111 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:54702
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:35995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.123111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:40.462615 123111 master_main.cc:80] Initializing master server...
I0227 17:25:40.463322 123111 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 221275us
I0227 17:25:40.464601 123111 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:40.464656 123111 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:40.464953 123111 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:40.464977 123111 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:40.467353 123111 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/instance:
uuid: "c31e774156b64c02922906e3e2985c89"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.467706 123111 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal/instance:
uuid: "c31e774156b64c02922906e3e2985c89"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.476575 123111 fs_manager.cc:503] Time spent creating directory manager: real 0.009s	user 0.007s	sys 0.000s
I0227 17:25:40.477306 123111 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:40.477731 123111 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:40.479918 123111 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:40.480000 123111 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data to 2721 blocks
I0227 17:25:40.481854 123111 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I0227 17:25:40.481945 123111 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
uuid: "c31e774156b64c02922906e3e2985c89"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.482054 123111 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:40.507030 123111 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:40.507936 123111 master_main.cc:83] Starting Master server...
I0227 17:25:40.514935 123111 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.254:35995
I0227 17:25:40.515359 123111 webserver.cc:175] Starting webserver on 127.112.245.254:0
I0227 17:25:40.515379 123111 webserver.cc:186] Document root disabled
I0227 17:25:40.516185 123111 webserver.cc:313] Webserver started. Bound to: http://127.112.245.254:45589/
I0227 17:25:40.516675 123111 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
I0227 17:25:40.519451 123540 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:40.520818 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 123111
I0227 17:25:40.521409 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:50732
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--hive_metastore_uris=thrift://127.0.0.1:54702 with env {}
I0227 17:25:40.522861 123540 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:40.532893 123540 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:25:40.533769 123540 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:50732: Network error: Client connection negotiation failed: client connection to 127.112.245.253:50732: connect: Connection refused (error 111)
I0227 17:25:40.553005 123540 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } attempt: 1
W0227 17:25:40.553722 123540 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:50732: Network error: Client connection negotiation failed: client connection to 127.112.245.253:50732: connect: Connection refused (error 111)
I0227 17:25:40.601862 123540 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } attempt: 2
W0227 17:25:40.602582 123540 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.253:50732: Network error: Client connection negotiation failed: client connection to 127.112.245.253:50732: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:40.607115 123549 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:40.607379 123549 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:40.608268 123549 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:54702
W0227 17:25:40.608285 123549 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:25:40.608295 123549 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:40.608302 123549 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:25:40.608311 123549 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:25:40.610088 123549 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:54702
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:50732
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.123549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:40.620836 123549 master_main.cc:80] Initializing master server...
I0227 17:25:40.621665 123549 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 221275us
I0227 17:25:40.623150 123549 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:40.623198 123549 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:40.623448 123549 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:40.623471 123549 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:40.626302 123549 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/instance:
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.626731 123549 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal/instance:
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.636683 123549 fs_manager.cc:503] Time spent creating directory manager: real 0.010s	user 0.008s	sys 0.000s
I0227 17:25:40.637563 123549 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:40.638087 123549 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:40.640919 123549 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:40.641006 123549 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data to 2721 blocks
I0227 17:25:40.643139 123549 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I0227 17:25:40.643240 123549 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.643337 123549 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:40.679131 123549 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:40.679994 123549 master_main.cc:83] Starting Master server...
I0227 17:25:40.686275 123549 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.253:50732
I0227 17:25:40.686797 123549 webserver.cc:175] Starting webserver on 127.112.245.253:0
I0227 17:25:40.686861 123549 webserver.cc:186] Document root disabled
I0227 17:25:40.687593 123549 webserver.cc:313] Webserver started. Bound to: http://127.112.245.253:58679/
I0227 17:25:40.688290 123549 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
I0227 17:25:40.689751 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 123549
I0227 17:25:40.690407 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:58453
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--hive_metastore_uris=thrift://127.0.0.1:54702 with env {}
I0227 17:25:40.691819 123914 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:40.696909 123914 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:40.713718 123540 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } attempt: 3
I0227 17:25:40.724232 123914 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:40.733851 123540 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:25:40.737119 123540 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:58453: Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
I0227 17:25:40.745661 123914 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } has no permanent_uuid. Determining permanent_uuid...
W0227 17:25:40.746580 123914 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:58453: Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
I0227 17:25:40.768052 123914 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } attempt: 1
W0227 17:25:40.768849 123914 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:58453: Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
I0227 17:25:40.801266 123540 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } attempt: 1
W0227 17:25:40.802161 123540 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:58453: Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:40.809895 123923 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:40.810333 123923 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:40.811579 123923 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:54702
W0227 17:25:40.811676 123923 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:25:40.811758 123923 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:40.811831 123923 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:25:40.811908 123923 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:25:40.814424 123923 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:54702
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:58453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.123923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:40.817003 123914 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } attempt: 2
W0227 17:25:40.817661 123914 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:58453: Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
I0227 17:25:40.825716 123923 master_main.cc:80] Initializing master server...
I0227 17:25:40.826687 123923 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 221275us
I0227 17:25:40.828420 123923 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:40.828541 123923 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:40.828915 123923 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:40.828999 123923 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:40.832034 123923 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/instance:
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.832516 123923 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal/instance:
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.854944 123923 fs_manager.cc:503] Time spent creating directory manager: real 0.022s	user 0.005s	sys 0.004s
I0227 17:25:40.857352 123923 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:40.858433 123923 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:40.867555 123923 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:40.867807 123923 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:25:40.871143 123923 fs_manager.cc:419] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I0227 17:25:40.871251 123923 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:40.871371 123923 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:40.881300 123540 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } attempt: 2
W0227 17:25:40.881916 123540 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.112.245.252:58453: Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
I0227 17:25:40.917177 123923 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:40.918131 123923 master_main.cc:83] Starting Master server...
I0227 17:25:40.925668 123923 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.252:58453
I0227 17:25:40.926141 123923 webserver.cc:175] Starting webserver on 127.112.245.252:0
I0227 17:25:40.926160 123923 webserver.cc:186] Document root disabled
I0227 17:25:40.926847 123923 webserver.cc:313] Webserver started. Bound to: http://127.112.245.252:52424/
I0227 17:25:40.927515 123923 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
I0227 17:25:40.929139 123914 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } attempt: 3
I0227 17:25:40.929806 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 123923
I0227 17:25:40.930526 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.112.245.193:0
--local_ip_for_outbound_sockets=127.112.245.193
--webserver_interface=127.112.245.193
--webserver_port=0
--tserver_master_addrs=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:25:40.934895 124241 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:40.941262 124241 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:40.955468 124241 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:40.956290 123914 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Bootstrap starting.
I0227 17:25:40.958789 123914 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: No blocks or log segments found. Creating new log.
I0227 17:25:40.959337 124241 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } has no permanent_uuid. Determining permanent_uuid...
I0227 17:25:40.960444 123914 log.cc:525] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:40.967314 124241 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Bootstrap starting.
I0227 17:25:40.968044 123540 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } attempt: 3
I0227 17:25:40.969907 124241 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: No blocks or log segments found. Creating new log.
I0227 17:25:40.971549 124241 log.cc:525] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:40.971599 123914 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: No bootstrap required, opened a new log
I0227 17:25:40.972947 124241 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: No bootstrap required, opened a new log
I0227 17:25:40.975175 123914 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:40.975246 123914 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:40.975296 123914 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Initialized, Role: FOLLOWER
I0227 17:25:40.975769 123914 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:40.976110 123914 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:40.976136 123914 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:40.976171 124241 raft_consensus.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:40.976253 124241 raft_consensus.cc:366] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:40.976302 124241 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3d7e621b864672aaa91ea2d0bbd1b9, State: Initialized, Role: FOLLOWER
I0227 17:25:40.977103 123914 sys_catalog.cc:424] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:25:40.977260 124241 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:40.977623 124241 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:40.977686 124241 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:40.978382 123549 master_main.cc:86] Master server successfully started.
I0227 17:25:40.978888 124241 sys_catalog.cc:424] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:25:40.980185 123923 master_main.cc:86] Master server successfully started.
I0227 17:25:40.980082 124302 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 2e3d7e621b864672aaa91ea2d0bbd1b9, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:40.980311 124294 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:40.980345 124294 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:40.980989 123540 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Bootstrap starting.
I0227 17:25:40.983284 123540 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: No blocks or log segments found. Creating new log.
W0227 17:25:40.984097 124304 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:25:40.984138 124304 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:25:40.984457 123540 log.cc:525] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:40.985648 124299 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 6cd5de61888340e681a3e5cd19fad418, Raft Consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:40.985689 124297 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:40.985715 124297 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:25:40.988350 124300 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:25:40.988390 124300 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I0227 17:25:41.001360 123540 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: No bootstrap required, opened a new log
I0227 17:25:41.004709 123540 raft_consensus.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.004781 123540 raft_consensus.cc:366] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:41.004832 123540 raft_consensus.cc:705] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31e774156b64c02922906e3e2985c89, State: Initialized, Role: FOLLOWER
I0227 17:25:41.005298 123540 consensus_queue.cc:229] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.005647 123540 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.005676 123540 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:41.005731 124326 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.005779 124326 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:41.008122 123540 sys_catalog.cc:424] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: configured and running, proceeding with master startup.
W0227 17:25:41.008527 123488 tablet.cc:1786] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:25:41.009654 124330 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0227 17:25:41.010051 123111 master_main.cc:86] Master server successfully started.
W0227 17:25:41.013232 124331 catalog_manager.cc:1104] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W0227 17:25:41.013265 124331 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W0227 17:25:41.168630 124192 tablet.cc:1786] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:25:41.180645 123827 tablet.cc:1786] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Can't schedule compaction. Clean time has not been advanced past its initial value.
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:41.187506 124249 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:41.188414 124249 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:41.205391 124249 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:41.205842 124249 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.193
I0227 17:25:41.214148 124249 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.112.245.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.124249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.193
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:41.219537 124294 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:41.219735 124294 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.221024 124294 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:41.221879 123515 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:41.221879 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:41.222100 124217 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 0.
I0227 17:25:41.222115 123515 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 0.
I0227 17:25:41.222872 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418; no voters: 
I0227 17:25:41.223182 124294 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:25:41.223227 124294 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:41.223255 124294 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:41.224514 124294 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.224936 124294 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 1 election: Requested vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:41.225324 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
I0227 17:25:41.225399 124217 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:41.225419 123515 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89"
I0227 17:25:41.225472 123515 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:41.226680 123515 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 1.
I0227 17:25:41.226711 124217 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 1.
I0227 17:25:41.227429 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:25:41.227577 124294 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:25:41.227798 124294 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 LEADER]: Becoming Leader. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Running, Role: LEADER
I0227 17:25:41.228230 124294 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.237872 124636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd5de61888340e681a3e5cd19fad418. Latest consensus state: current_term: 1 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.237908 124636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:41.238251 124694 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:25:41.240787 124694 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:25:41.245507 124249 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:25:41.252063 124249 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 221775us
I0227 17:25:41.254709 124249 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:41.254920 124249 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:41.255985 124249 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:41.256193 124249 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:41.265074 124249 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data/instance:
uuid: "9f6459ebb80643b98e01844aa2fd6338"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.266623 124249 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/wal/instance:
uuid: "9f6459ebb80643b98e01844aa2fd6338"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.269245 123515 raft_consensus.cc:1181] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 1 FOLLOWER]: Refusing update from remote peer 6cd5de61888340e681a3e5cd19fad418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:25:41.269279 124217 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 1 FOLLOWER]: Refusing update from remote peer 6cd5de61888340e681a3e5cd19fad418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I0227 17:25:41.270125 124636 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:41.270416 124294 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:41.274125 124326 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd5de61888340e681a3e5cd19fad418. Latest consensus state: current_term: 1 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.274159 124326 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:41.276545 124297 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd5de61888340e681a3e5cd19fad418. Latest consensus state: current_term: 1 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.276579 124297 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:41.279260 124294 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.279299 124294 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:41.279405 124714 mvcc.cc:203] Tried to move safe_time back from 6354195010630582272 to 6354195010473553920. Current Snapshot: MvccSnapshot[committed={T|T < 6354195010630582272}]
I0227 17:25:41.280982 124297 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.281018 124297 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:41.284732 124249 fs_manager.cc:503] Time spent creating directory manager: real 0.017s	user 0.005s	sys 0.003s
I0227 17:25:41.285346 124636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.285380 124636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:41.285580 124249 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:41.286103 124249 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:41.286895 124326 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.286923 124326 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:41.287091 124694 catalog_manager.cc:941] Generated new certificate authority record
I0227 17:25:41.288938 124694 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:25:41.289057 124249 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:41.289142 124249 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data/data to 2721 blocks
I0227 17:25:41.291647 124249 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.001s
I0227 17:25:41.291776 124249 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/wal
uuid: "9f6459ebb80643b98e01844aa2fd6338"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.291900 124249 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:41.312129 124694 catalog_manager.cc:4095] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Generated new TSK 0
I0227 17:25:41.312705 124694 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:25:41.322196 124249 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:41.323714 124249 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:25:41.323855 124249 tablet_server_main.cc:90] Starting tablet server...
I0227 17:25:41.341648 124249 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.193:55806
I0227 17:25:41.342032 124249 webserver.cc:175] Starting webserver on 127.112.245.193:0
I0227 17:25:41.342046 124249 webserver.cc:186] Document root disabled
I0227 17:25:41.342674 124249 webserver.cc:313] Webserver started. Bound to: http://127.112.245.193:47497/
I0227 17:25:41.343227 124249 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-0/data/info.pb
I0227 17:25:41.344616 124249 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:25:41.345415 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 124249
I0227 17:25:41.346058 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data
--block_manager=log
--rpc_bind_addresses=127.112.245.194:0
--local_ip_for_outbound_sockets=127.112.245.194
--webserver_interface=127.112.245.194
--webserver_port=0
--tserver_master_addrs=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:25:41.378249 124856 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:25:41.378406 124856 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.378881 124856 heartbeater.cc:474] Master 127.112.245.253:50732 requested a full tablet report, sending...
I0227 17:25:41.380219 123865 ts_manager.cc:98] Registered new tserver with Master: 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:25:41.381788 124857 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:25:41.381840 124857 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.381986 123865 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.193:50522
I0227 17:25:41.381994 124857 heartbeater.cc:474] Master 127.112.245.254:35995 requested a full tablet report, sending...
I0227 17:25:41.385262 124855 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:25:41.385313 124855 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.385460 124855 heartbeater.cc:474] Master 127.112.245.252:58453 requested a full tablet report, sending...
I0227 17:25:41.386396 124199 ts_manager.cc:98] Registered new tserver with Master: 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:25:41.389016 123501 ts_manager.cc:98] Registered new tserver with Master: 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:41.447412 124859 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:41.447705 124859 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:41.448822 124859 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:41.448946 124859 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.194
I0227 17:25:41.451195 124859 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.112.245.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.124859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.194
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:41.461108 124859 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:25:41.465657 124859 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 221775us
I0227 17:25:41.467294 124859 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:41.467416 124859 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:41.468371 124859 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:41.468468 124859 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:41.471936 124859 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data/instance:
uuid: "39144add463f4fe89dc1ea42c778d8cf"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.472450 124859 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/wal/instance:
uuid: "39144add463f4fe89dc1ea42c778d8cf"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.492970 124859 fs_manager.cc:503] Time spent creating directory manager: real 0.020s	user 0.004s	sys 0.003s
I0227 17:25:41.493839 124859 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:41.494343 124859 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:41.497557 124859 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:41.497694 124859 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data/data to 2721 blocks
I0227 17:25:41.500082 124859 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I0227 17:25:41.500195 124859 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/wal
uuid: "39144add463f4fe89dc1ea42c778d8cf"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.500306 124859 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:41.569478 124859 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:41.571087 124859 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:25:41.571305 124859 tablet_server_main.cc:90] Starting tablet server...
I0227 17:25:41.588323 124859 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.194:55958
I0227 17:25:41.588732 124859 webserver.cc:175] Starting webserver on 127.112.245.194:0
I0227 17:25:41.588762 124859 webserver.cc:186] Document root disabled
I0227 17:25:41.589537 124859 webserver.cc:313] Webserver started. Bound to: http://127.112.245.194:50850/
I0227 17:25:41.590121 124859 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-1/data/info.pb
I0227 17:25:41.591266 124859 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:25:41.599229 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 124859
I0227 17:25:41.599897 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data
--block_manager=log
--rpc_bind_addresses=127.112.245.195:0
--local_ip_for_outbound_sockets=127.112.245.195
--webserver_interface=127.112.245.195
--webserver_port=0
--tserver_master_addrs=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I0227 17:25:41.623970 125068 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:25:41.624116 125068 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.624392 125069 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:25:41.624440 125069 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.624584 125069 heartbeater.cc:474] Master 127.112.245.253:50732 requested a full tablet report, sending...
I0227 17:25:41.624581 125068 heartbeater.cc:474] Master 127.112.245.252:58453 requested a full tablet report, sending...
I0227 17:25:41.625288 125070 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:25:41.625334 125070 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.625445 125070 heartbeater.cc:474] Master 127.112.245.254:35995 requested a full tablet report, sending...
I0227 17:25:41.625587 124199 ts_manager.cc:98] Registered new tserver with Master: 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:25:41.625830 123865 ts_manager.cc:98] Registered new tserver with Master: 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:25:41.626458 123501 ts_manager.cc:98] Registered new tserver with Master: 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:25:41.626849 123865 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.194:49670
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:41.728651 125074 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:41.728919 125074 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:41.729964 125074 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:41.730134 125074 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.112.245.195
I0227 17:25:41.735008 125074 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.112.245.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.112.245.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--never_fsync=true
--heap_profile_path=/tmp/kudu-tserver.125074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.112.245.195
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:41.745518 125074 tablet_server_main.cc:85] Initializing tablet server...
I0227 17:25:41.747645 125074 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 221775us
I0227 17:25:41.748493 125074 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:41.748600 125074 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:41.748936 125074 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I0227 17:25:41.749012 125074 server_base.cc:433] Attempting to create new FS layout instead
I0227 17:25:41.751834 125074 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data/instance:
uuid: "9a4f55686db84876a066f455f6a41180"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.752329 125074 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/wal/instance:
uuid: "9a4f55686db84876a066f455f6a41180"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.782002 125074 fs_manager.cc:503] Time spent creating directory manager: real 0.029s	user 0.004s	sys 0.003s
I0227 17:25:41.785418 125074 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:41.787930 125074 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:41.791909 125074 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:41.797888 125074 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data/data to 2721 blocks
I0227 17:25:41.800094 125074 fs_manager.cc:419] Time spent opening block manager: real 0.008s	user 0.001s	sys 0.001s
I0227 17:25:41.800330 125074 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/wal
uuid: "9a4f55686db84876a066f455f6a41180"
format_stamp: "Formatted at 2019-02-28 01:25:41 on ve0518.halxg.cloudera.com"
I0227 17:25:41.800523 125074 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:41.812952 125246 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:25:41.813172 125246 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.814291 125246 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:41.814672 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:41.814751 124217 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 1.
I0227 17:25:41.814797 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:41.814927 123878 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:41.815516 123455 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:25:41.815768 125246 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 1 FOLLOWER]: Leader pre-election won for term 2
I0227 17:25:41.815847 125246 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:25:41.815923 125246 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:25:41.817235 125246 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.817699 125246 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 2 election: Requested vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:41.817912 125247 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:25:41.818011 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418"
I0227 17:25:41.818069 123878 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 LEADER]: Leader election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:41.818003 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
I0227 17:25:41.820291 125247 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.821408 124217 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:25:41.822479 124217 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 2.
I0227 17:25:41.823071 123455 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89; no voters: 6cd5de61888340e681a3e5cd19fad418
I0227 17:25:41.823571 125247 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:25:41.823701 123515 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:41.823776 123515 raft_consensus.cc:2276] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in current term 2: Already voted for candidate c31e774156b64c02922906e3e2985c89 in this term.
I0227 17:25:41.823829 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:41.823887 123878 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 2 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:41.824065 125246 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 2 FOLLOWER]: Leader election won for term 2
I0227 17:25:41.824316 124105 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:25:41.824614 125246 raft_consensus.cc:667] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 2 LEADER]: Becoming Leader. State: Replica: c31e774156b64c02922906e3e2985c89, State: Running, Role: LEADER
I0227 17:25:41.824640 125247 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I0227 17:25:41.824968 125246 consensus_queue.cc:206] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.825888 125246 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 2 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:41.825964 125246 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:41.826249 125255 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:25:41.827764 125255 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:25:41.830284 125255 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:25:41.831132 125255 catalog_manager.cc:4106] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Loaded TSK: 0
I0227 17:25:41.832017 125255 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:25:41.845981 125074 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:41.847512 125074 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I0227 17:25:41.847723 125074 tablet_server_main.cc:90] Starting tablet server...
I0227 17:25:41.857849 125074 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.195:45942
I0227 17:25:41.858181 125074 webserver.cc:175] Starting webserver on 127.112.245.195:0
I0227 17:25:41.858255 125074 webserver.cc:186] Document root disabled
I0227 17:25:41.858834 125074 webserver.cc:313] Webserver started. Bound to: http://127.112.245.195:46354/
I0227 17:25:41.859460 125074 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/ts-2/data/info.pb
I0227 17:25:41.860357 125074 tablet_server_main.cc:93] Tablet server successfully started.
I0227 17:25:41.868477 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver as pid 125074
I0227 17:25:41.870394 125383 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:25:41.870506 125383 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.870893 125383 heartbeater.cc:474] Master 127.112.245.253:50732 requested a full tablet report, sending...
I0227 17:25:41.871893 123865 ts_manager.cc:98] Registered new tserver with Master: 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:25:41.872597 125382 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:25:41.872664 125382 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.872844 123865 master_service.cc:263] Signed X509 certificate for tserver {username='adar'} at 127.112.245.195:35950
I0227 17:25:41.872879 125382 heartbeater.cc:474] Master 127.112.245.252:58453 requested a full tablet report, sending...
I0227 17:25:41.873994 124199 ts_manager.cc:98] Registered new tserver with Master: 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:25:41.879321 125384 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:25:41.879374 125384 heartbeater.cc:425] Registering TS with master...
I0227 17:25:41.879508 125384 heartbeater.cc:474] Master 127.112.245.254:35995 requested a full tablet report, sending...
I0227 17:25:41.882141 123501 ts_manager.cc:98] Registered new tserver with Master: 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:25:41.884624 124856 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
I0227 17:25:41.887485 115671 external_mini_cluster.cc:623] 3 TS(s) registered with all masters
I0227 17:25:41.900934 124217 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Rejecting Update request from peer 6cd5de61888340e681a3e5cd19fad418 for earlier term 1. Current term is 2. Ops: []
I0227 17:25:41.901377 125412 consensus_queue.cc:995] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I0227 17:25:41.901588 125412 raft_consensus.cc:2881] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 LEADER]: Stepping down as leader of term 1
I0227 17:25:41.901609 125412 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Running, Role: LEADER
I0227 17:25:41.901741 123515 raft_consensus.cc:1149] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 2 LEADER]: Rejecting Update request from peer 6cd5de61888340e681a3e5cd19fad418 for earlier term 1. Current term is 2. Ops: []
I0227 17:25:41.901804 125412 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:41.901834 125412 consensus_peers.cc:482] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 -> Peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Closing peer: 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:25:41.901854 125412 consensus_peers.cc:482] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 -> Peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Closing peer: c31e774156b64c02922906e3e2985c89
I0227 17:25:41.901983 125412 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 1 FOLLOWER]: Advancing to term 2
I0227 17:25:41.906425 115671 test_util.cc:212] Using random seed: 724394998
I0227 17:25:41.906590 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 123111
W0227 17:25:41.927716 124741 connection.cc:647] client connection to 127.112.245.254:35995 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.254:35995): Connection reset by peer (error 104)
W0227 17:25:41.927829 125257 connection.cc:647] client connection to 127.112.245.254:35995 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.254:35995): Connection reset by peer (error 104)
W0227 17:25:41.927829 124930 connection.cc:647] client connection to 127.112.245.254:35995 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.254:35995): Connection reset by peer (error 104)
I0227 17:25:41.987500 124304 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: acquiring CA information for follower catalog manager: success
I0227 17:25:41.988760 124304 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:25:41.990106 124300 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:25:42.079044 125475 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:42.079356 125475 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
W0227 17:25:42.082053 125475 data_dirs.cc:617] IO error: Could not lock block_manager_instance file. Make sure that Kudu is not already running and you are not trying to run Kudu with a different user than before: lock /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data/block_manager_instance: Resource temporarily unavailable (error 11)
W0227 17:25:42.082151 125475 data_dirs.cc:618] Proceeding without lock
I0227 17:25:42.090804 125475 fs_manager.cc:399] Time spent opening directory manager: real 0.010s	user 0.004s	sys 0.002s
I0227 17:25:42.091512 125475 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:42.092308 125475 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:42.095659 125475 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:42.095815 125475 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:25:42.096760 125678 log_block_manager.cc:2649] Read-only block manager, skipping repair
I0227 17:25:42.097666 125475 fs_report.cc:352] FS layout report
--------------------
wal directory: 
metadata directory: 
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:42.097841 125475 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I0227 17:25:42.098356 125475 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:42.120402 115671 master_failover-itest.cc:410] UUIDS: c31e774156b64c02922906e3e2985c89 6cd5de61888340e681a3e5cd19fad418 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:25:42.128597 125069 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
W0227 17:25:42.129349 125070 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:35995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.112.245.254:35995: connect: Connection refused (error 111)
I0227 17:25:42.262112 125687 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:42.262418 125687 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:42.268178 125687 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/instance:
uuid: "c31e774156b64c02922906e3e2985c89"
format_stamp: "Formatted at 2019-02-28 01:25:42 on ve0518.halxg.cloudera.com"
I0227 17:25:42.271114 125687 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal/instance:
uuid: "c31e774156b64c02922906e3e2985c89"
format_stamp: "Formatted at 2019-02-28 01:25:42 on ve0518.halxg.cloudera.com"
I0227 17:25:42.292256 125687 fs_manager.cc:503] Time spent creating directory manager: real 0.021s	user 0.006s	sys 0.004s
I0227 17:25:42.374889 125383 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
I0227 17:25:42.383307 125384 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
W0227 17:25:42.390441 125384 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:35995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.112.245.254:35995: connect: Connection refused (error 111)
I0227 17:25:42.391453 124857 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:25:42.397070 125877 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:42.397199 125877 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:42.397692 125877 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:25:42.398082 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:42.398195 123878 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 2.
I0227 17:25:42.398512 124105 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418; no voters: 
I0227 17:25:42.399075 125877 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I0227 17:25:42.399137 125877 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:42.399169 125877 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 2 FOLLOWER]: Advancing to term 3
W0227 17:25:42.400012 124857 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:35995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.112.245.254:35995: connect: Connection refused (error 111)
I0227 17:25:42.400343 125877 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:42.400746 125877 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 3 election: Requested vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:25:42.401046 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418"
I0227 17:25:42.401109 123878 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 2 FOLLOWER]: Advancing to term 3
I0227 17:25:42.402222 123878 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 3.
I0227 17:25:42.402503 124105 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418; no voters: 
I0227 17:25:42.406947 125877 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 FOLLOWER]: Leader election won for term 3
I0227 17:25:42.407109 125877 raft_consensus.cc:667] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Becoming Leader. State: Replica: 2e3d7e621b864672aaa91ea2d0bbd1b9, State: Running, Role: LEADER
I0227 17:25:42.407444 125877 consensus_queue.cc:206] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:42.408411 125877 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3d7e621b864672aaa91ea2d0bbd1b9. Latest consensus state: current_term: 3 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:42.408438 125877 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: LEADER
W0227 17:25:42.408880 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Network error: Client connection negotiation failed: client connection to 127.112.245.254:35995: connect: Connection refused (error 111)
W0227 17:25:42.408990 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Network error: Client connection negotiation failed: client connection to 127.112.245.254:35995: connect: Connection refused (error 111)
I0227 17:25:42.419277 125896 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:25:42.420610 125896 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:25:42.421413 125896 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:25:42.421941 125896 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Loaded TSK: 0
I0227 17:25:42.422621 125896 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:25:42.516993 125838 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:42.517271 125838 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:42.531630 125838 fs_manager.cc:399] Time spent opening directory manager: real 0.012s	user 0.004s	sys 0.001s
I0227 17:25:42.532585 125838 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:42.533444 125838 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:42.536383 125838 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:42.536556 125838 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data to 2721 blocks
I0227 17:25:42.540563 125838 fs_report.cc:352] FS layout report
--------------------
wal directory: 
metadata directory: 
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:42.540781 125838 fs_manager.cc:419] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.001s
I0227 17:25:42.541373 125838 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
uuid: "c31e774156b64c02922906e3e2985c89"
format_stamp: "Formatted at 2019-02-28 01:25:42 on ve0518.halxg.cloudera.com"
I0227 17:25:42.566853 125838 tablet_copy_client.cc:240] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: tablet copy: Beginning tablet copy session from remote peer at address 127.112.245.252:58453
I0227 17:25:42.618574 124222 tablet_copy_service.cc:135] P 2e3d7e621b864672aaa91ea2d0bbd1b9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c31e774156b64c02922906e3e2985c89 ({username='adar'} at 127.112.245.252:49743)
I0227 17:25:42.618672 124222 tablet_copy_service.cc:156] P 2e3d7e621b864672aaa91ea2d0bbd1b9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c31e774156b64c02922906e3e2985c89 at {username='adar'} at 127.112.245.252:49743: session id = c31e774156b64c02922906e3e2985c89-00000000000000000000000000000000
I0227 17:25:42.620218 124222 tablet_copy_source_session.cc:202] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Tablet Copy: opened 0 blocks and 1 log segments
I0227 17:25:42.627256 125838 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I0227 17:25:42.641676 125838 tablet_copy_client.cc:575] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: tablet copy: Starting download of 0 data blocks...
I0227 17:25:42.642742 125838 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: tablet copy: Starting download of 1 WAL segments...
I0227 17:25:42.657411 125838 tablet_copy_client.cc:414] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I0227 17:25:42.693542 124222 tablet_copy_service.cc:325] P 2e3d7e621b864672aaa91ea2d0bbd1b9: Request end of tablet copy session c31e774156b64c02922906e3e2985c89-00000000000000000000000000000000 received from {username='adar'} at 127.112.245.252:49743
I0227 17:25:42.693603 124222 tablet_copy_service.cc:413] P 2e3d7e621b864672aaa91ea2d0bbd1b9: ending tablet copy session c31e774156b64c02922906e3e2985c89-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c31e774156b64c02922906e3e2985c89
I0227 17:25:42.740619 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:35995
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--hive_metastore_uris=thrift://127.0.0.1:54702 with env {}
W0227 17:25:42.785571 124104 consensus_peers.cc:457] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 -> Peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Couldn't send request to peer c31e774156b64c02922906e3e2985c89. Status: Network error: Client connection negotiation failed: client connection to 127.112.245.254:35995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:25:42.964700 126213 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:25:42.965137 126213 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:25:42.966516 126213 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:54702
W0227 17:25:42.968169 126213 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:25:42.968266 126213 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:25:42.968348 126213 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:25:42.968427 126213 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:25:42.981472 126213 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:54702
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.254:35995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
--webserver_interface=127.112.245.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.126213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:25:42.994297 126213 master_main.cc:80] Initializing master server...
I0227 17:25:43.001021 126213 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 222275us
I0227 17:25:43.002825 126213 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:43.002966 126213 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:43.009666 126213 fs_manager.cc:399] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
I0227 17:25:43.010277 126213 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:43.010869 126213 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:43.013892 126213 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:43.014125 126213 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data to 2721 blocks
I0227 17:25:43.016247 126213 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I0227 17:25:43.017165 126213 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
uuid: "c31e774156b64c02922906e3e2985c89"
format_stamp: "Formatted at 2019-02-28 01:25:42 on ve0518.halxg.cloudera.com"
I0227 17:25:43.017356 126213 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:43.024052 123878 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Refusing update from remote peer 2e3d7e621b864672aaa91ea2d0bbd1b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I0227 17:25:43.024930 125893 consensus_queue.cc:984] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I0227 17:25:43.035689 126583 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3d7e621b864672aaa91ea2d0bbd1b9. Latest consensus state: current_term: 3 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.035769 126583 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:43.036870 125893 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.036909 125893 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:43.037788 126583 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.037875 126583 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:43.100538 126213 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:25:43.101517 126213 master_main.cc:83] Starting Master server...
I0227 17:25:43.115762 126213 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.254:35995
I0227 17:25:43.116266 126213 webserver.cc:175] Starting webserver on 127.112.245.254:0
I0227 17:25:43.116292 126213 webserver.cc:186] Document root disabled
I0227 17:25:43.116988 126213 webserver.cc:313] Webserver started. Bound to: http://127.112.245.254:46298/
I0227 17:25:43.117786 126213 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-0/data/info.pb
I0227 17:25:43.126440 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 126213
I0227 17:25:43.130424 125068 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
I0227 17:25:43.133992 126695 sys_catalog.cc:188] Verifying existing consensus state
I0227 17:25:43.135874 126695 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Bootstrap starting.
I0227 17:25:43.145750 126695 log.cc:525] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Log is configured to *not* fsync() on all Append() calls
W0227 17:25:43.161227 125414 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.254:35995)
I0227 17:25:43.161662 125070 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:25:43.165743 126695 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I0227 17:25:43.166316 126695 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Bootstrap complete.
I0227 17:25:43.170697 126695 raft_consensus.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:43.180539 126695 raft_consensus.cc:705] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31e774156b64c02922906e3e2985c89, State: Initialized, Role: FOLLOWER
I0227 17:25:43.181152 126695 consensus_queue.cc:229] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:43.181551 126695 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.181643 126695 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:43.182540 126695 sys_catalog.cc:424] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:25:43.184279 126213 master_main.cc:86] Master server successfully started.
I0227 17:25:43.189400 126734 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.189478 126734 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:43.210414 126738 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: c31e774156b64c02922906e3e2985c89, Raft Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.218684 126739 catalog_manager.cc:1102] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: acquiring CA information for follower catalog manager: success
I0227 17:25:43.228389 126739 catalog_manager.cc:1130] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:25:43.272792 126660 raft_consensus.cc:1126] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[3.4-3.4]   Dedup: 3.4->[]
I0227 17:25:43.274179 126734 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3d7e621b864672aaa91ea2d0bbd1b9. Latest consensus state: current_term: 3 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.274209 126734 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:43.274328 126734 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.274348 126734 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:43.378785 125382 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
I0227 17:25:43.392731 124855 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
I0227 17:25:43.412281 125384 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:25:43.412376 125384 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:25:43.413477 126651 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "9a4f55686db84876a066f455f6a41180" instance_seqno: 1551317141855821) as {username='adar'} at 127.112.245.195:35483; Asking this server to re-register.
I0227 17:25:43.413944 125384 heartbeater.cc:425] Registering TS with master...
I0227 17:25:43.414829 126651 ts_manager.cc:98] Registered new tserver with Master: 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:25:43.430532 124857 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:25:43.430634 124857 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:25:43.431326 126651 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" instance_seqno: 1551317141335050) as {username='adar'} at 127.112.245.193:36811; Asking this server to re-register.
I0227 17:25:43.431784 124857 heartbeater.cc:425] Registering TS with master...
I0227 17:25:43.432540 126651 ts_manager.cc:98] Registered new tserver with Master: 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:25:43.537482 126583 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:43.537598 126583 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:43.538072 126583 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:43.540500 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:43.540587 124217 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:43.556704 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:43.556872 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:43.557369 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:43.557567 126583 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I0227 17:25:43.632326 124199 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.252:49654:
name: "default.table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:25:43.633510 124199 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:25:43.664131 125070 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:25:43.664871 126651 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" instance_seqno: 1551317141580643) as {username='adar'} at 127.112.245.194:54775; Asking this server to re-register.
I0227 17:25:43.665346 125070 heartbeater.cc:425] Registering TS with master...
I0227 17:25:43.666088 126651 ts_manager.cc:98] Registered new tserver with Master: 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:25:43.774103 126827 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:43.774286 126827 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:43.775773 126827 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:43.796139 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:43.796211 124217 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:43.800129 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:43.800207 123878 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:43.800624 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:25:43.800942 126827 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I0227 17:25:43.833357 126586 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:43.833406 126586 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:44.736014 127918 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:44.736179 127918 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:44.736750 127918 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:44.737181 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:44.737267 124217 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:44.739333 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:44.739413 123878 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:44.739748 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:25:44.740711 127918 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W0227 17:25:45.001847 124301 hms_client.cc:245] Time spent create HMS table: real 1.367s	user 0.000s	sys 0.000s
I0227 17:25:45.033823 124199 rpcz_store.cc:269] Call kudu.master.MasterService.CreateTable from 127.112.245.252:49654 (request call id 3) took 1869ms. Request Metrics: {"Hive Metastore.queue_time_us":38,"Hive Metastore.run_cpu_time_us":480,"Hive Metastore.run_wall_time_us":1367463,"child_traces":[["txn",{"apply.queue_time_us":4756,"apply.run_cpu_time_us":470,"apply.run_wall_time_us":470,"num_ops":2,"prepare.queue_time_us":8180,"prepare.run_cpu_time_us":932,"prepare.run_wall_time_us":956,"raft.queue_time_us":342,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":357,"replication_time_us":16233,"thread_start_us":337,"threads_started":4,"wal-append.queue_time_us":348}]]}
I0227 17:25:45.058331 124988 tablet_service.cc:774] Processing CreateTablet for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table=default.table_0 [id=bc75674adf8e42de8857bcba6d648867]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:45.058802 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fcd66b3d06b5442b9b1b4d34c6d0667b. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:45.058862 125316 tablet_service.cc:774] Processing CreateTablet for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table=default.table_0 [id=bc75674adf8e42de8857bcba6d648867]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:45.059332 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fcd66b3d06b5442b9b1b4d34c6d0667b. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:45.063377 124778 tablet_service.cc:774] Processing CreateTablet for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table=default.table_0 [id=bc75674adf8e42de8857bcba6d648867]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:45.063796 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fcd66b3d06b5442b9b1b4d34c6d0667b. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:45.064918 125316 ts_tablet_manager.cc:1175] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:45.065986 124988 ts_tablet_manager.cc:1175] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:45.068195 124778 ts_tablet_manager.cc:1175] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:45.079288 128163 ts_tablet_manager.cc:1030] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:25:45.079685 128163 tablet_bootstrap.cc:439] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:25:45.080204 128167 ts_tablet_manager.cc:1030] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:25:45.080533 128167 tablet_bootstrap.cc:439] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:25:45.082074 128163 tablet_bootstrap.cc:593] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:25:45.082507 128167 tablet_bootstrap.cc:593] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:25:45.083112 128163 log.cc:525] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:45.083482 128167 log.cc:525] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:45.088153 128168 ts_tablet_manager.cc:1030] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:25:45.088490 128168 tablet_bootstrap.cc:439] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:25:45.091972 128163 tablet_bootstrap.cc:439] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:25:45.092206 128163 ts_tablet_manager.cc:1047] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.001s
I0227 17:25:45.095566 128163 raft_consensus.cc:340] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.095645 128163 raft_consensus.cc:366] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:45.095686 128163 raft_consensus.cc:705] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:25:45.096098 128163 consensus_queue.cc:229] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.096920 128163 ts_tablet_manager.cc:1075] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.002s
I0227 17:25:45.100244 128168 tablet_bootstrap.cc:593] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:25:45.100436 128167 tablet_bootstrap.cc:439] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:25:45.100646 128167 ts_tablet_manager.cc:1047] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.000s
I0227 17:25:45.101290 128168 log.cc:525] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: Log is configured to *not* fsync() on all Append() calls
I0227 17:25:45.103736 128167 raft_consensus.cc:340] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.103822 128167 raft_consensus.cc:366] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:45.103868 128167 raft_consensus.cc:705] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:25:45.104269 128167 consensus_queue.cc:229] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.105072 128167 ts_tablet_manager.cc:1075] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I0227 17:25:45.114174 128168 tablet_bootstrap.cc:439] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:25:45.114388 128168 ts_tablet_manager.cc:1047] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.026s	user 0.003s	sys 0.001s
I0227 17:25:45.117779 128168 raft_consensus.cc:340] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.117875 128168 raft_consensus.cc:366] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:45.117974 128168 raft_consensus.cc:705] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:25:45.118577 128168 consensus_queue.cc:229] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.119397 128168 ts_tablet_manager.cc:1075] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.001s
W0227 17:25:45.121935 125385 tablet.cc:1786] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:25:45.351511 125071 tablet.cc:1786] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:25:45.360224 124858 tablet.cc:1786] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:25:45.434473 128181 raft_consensus.cc:472] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:45.434659 128181 raft_consensus.cc:494] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.435791 128181 leader_election.cc:292] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:25:45.455413 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "fcd66b3d06b5442b9b1b4d34c6d0667b" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:25:45.455606 124804 raft_consensus.cc:2346] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:25:45.456208 124930 leader_election.cc:306] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:25:45.456539 128181 raft_consensus.cc:2640] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:25:45.456584 128181 raft_consensus.cc:472] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:45.456614 128181 raft_consensus.cc:2886] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:45.457943 128181 raft_consensus.cc:494] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.458441 128181 leader_election.cc:292] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Requested vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:25:45.458560 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "fcd66b3d06b5442b9b1b4d34c6d0667b" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:25:45.458736 125337 raft_consensus.cc:2346] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:25:45.459121 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "fcd66b3d06b5442b9b1b4d34c6d0667b" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:25:45.459184 124804 raft_consensus.cc:2886] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:45.460310 124804 raft_consensus.cc:2346] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:25:45.460717 124930 leader_election.cc:306] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:25:45.460940 128181 raft_consensus.cc:2640] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:25:45.461120 128181 raft_consensus.cc:667] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [term 1 LEADER]: Becoming Leader. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Running, Role: LEADER
I0227 17:25:45.461426 128181 consensus_queue.cc:206] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:25:45.464397 124199 catalog_manager.cc:3791] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194). New cstate: current_term: 1 leader_uuid: "39144add463f4fe89dc1ea42c778d8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } }
I0227 17:25:45.466488 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "fcd66b3d06b5442b9b1b4d34c6d0667b" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:25:45.466580 125337 raft_consensus.cc:2886] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:45.467911 125337 raft_consensus.cc:2346] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
W0227 17:25:45.535221 124966 tablet_service.cc:1915] Rejecting scan request for tablet fcd66b3d06b5442b9b1b4d34c6d0667b: Uninitialized: safe time has not yet been initialized
W0227 17:25:45.737212 124966 tablet_service.cc:1915] Rejecting scan request for tablet fcd66b3d06b5442b9b1b4d34c6d0667b: Uninitialized: safe time has not yet been initialized
I0227 17:25:45.954882 125337 raft_consensus.cc:1181] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:45.962908 128274 consensus_queue.cc:984] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:45.969584 128545 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:45.969820 128545 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:45.970713 128545 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:45.970829 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:45.970882 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:45.970906 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:45.970949 124217 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:45.971284 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:45.971518 128545 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I0227 17:25:45.979499 124804 raft_consensus.cc:1181] T fcd66b3d06b5442b9b1b4d34c6d0667b P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:45.982050 128549 consensus_queue.cc:984] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:46.185931 124199 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.252:49654:
name: "default.table_0_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:25:46.186295 124199 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:25:46.396658 124988 tablet_service.cc:774] Processing CreateTablet for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table=default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:46.396900 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9808ecd2a4be428290fa53bb51dd17d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:46.397436 125316 tablet_service.cc:774] Processing CreateTablet for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table=default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:46.397658 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9808ecd2a4be428290fa53bb51dd17d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:46.398851 124988 ts_tablet_manager.cc:1175] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:46.399159 124778 tablet_service.cc:774] Processing CreateTablet for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table=default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:46.399390 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9808ecd2a4be428290fa53bb51dd17d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:46.399519 125316 ts_tablet_manager.cc:1175] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:46.401242 124778 ts_tablet_manager.cc:1175] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:46.403381 128609 ts_tablet_manager.cc:1030] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:25:46.403703 128609 tablet_bootstrap.cc:439] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:25:46.406325 128609 tablet_bootstrap.cc:593] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:25:46.408337 128607 ts_tablet_manager.cc:1030] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:25:46.409322 128607 tablet_bootstrap.cc:439] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:25:46.410164 128608 ts_tablet_manager.cc:1030] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:25:46.410326 128608 tablet_bootstrap.cc:439] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:25:46.410557 128607 tablet_bootstrap.cc:593] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:25:46.411402 128608 tablet_bootstrap.cc:593] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:25:46.419831 128608 tablet_bootstrap.cc:439] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:25:46.419931 128608 ts_tablet_manager.cc:1047] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.001s
I0227 17:25:46.420675 128608 raft_consensus.cc:340] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.420708 128608 raft_consensus.cc:366] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:46.420729 128608 raft_consensus.cc:705] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:25:46.420871 128608 consensus_queue.cc:229] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.421136 128608 ts_tablet_manager.cc:1075] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:25:46.424892 128609 tablet_bootstrap.cc:439] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:25:46.425016 128609 ts_tablet_manager.cc:1047] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.021s	user 0.001s	sys 0.001s
I0227 17:25:46.426766 128609 raft_consensus.cc:340] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.427641 128609 raft_consensus.cc:366] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:46.427738 128609 raft_consensus.cc:705] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:25:46.428009 128609 consensus_queue.cc:229] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.429903 128547 raft_consensus.cc:472] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:46.430028 128547 raft_consensus.cc:494] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.430871 128607 tablet_bootstrap.cc:439] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:25:46.430992 128607 ts_tablet_manager.cc:1047] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.022s	user 0.001s	sys 0.001s
I0227 17:25:46.431180 128547 leader_election.cc:292] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:25:46.431787 128607 raft_consensus.cc:340] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.431864 128607 raft_consensus.cc:366] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:46.431910 128607 raft_consensus.cc:705] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:25:46.432081 128607 consensus_queue.cc:229] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.432634 128607 ts_tablet_manager.cc:1075] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:25:46.434334 128609 ts_tablet_manager.cc:1075] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I0227 17:25:46.451215 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "9808ecd2a4be428290fa53bb51dd17d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:25:46.451318 124804 raft_consensus.cc:2346] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:25:46.451858 125257 leader_election.cc:306] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4f55686db84876a066f455f6a41180, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:25:46.452172 128547 raft_consensus.cc:2640] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:25:46.452201 128547 raft_consensus.cc:472] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:46.452229 128547 raft_consensus.cc:2886] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:46.453470 128547 raft_consensus.cc:494] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.453981 128547 leader_election.cc:292] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Requested vote from peers 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:25:46.454164 125012 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "9808ecd2a4be428290fa53bb51dd17d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf" is_pre_election: true
I0227 17:25:46.454295 125012 raft_consensus.cc:2346] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:25:46.454566 125012 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "9808ecd2a4be428290fa53bb51dd17d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf"
I0227 17:25:46.454571 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "9808ecd2a4be428290fa53bb51dd17d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:25:46.454607 125012 raft_consensus.cc:2886] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:46.454630 124804 raft_consensus.cc:2886] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:46.455708 124804 raft_consensus.cc:2346] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:25:46.455739 125012 raft_consensus.cc:2346] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:25:46.456209 125257 leader_election.cc:306] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9a4f55686db84876a066f455f6a41180; no voters: 
I0227 17:25:46.456351 128547 raft_consensus.cc:2640] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:25:46.456543 128547 raft_consensus.cc:667] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Running, Role: LEADER
I0227 17:25:46.456861 128547 consensus_queue.cc:206] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:46.459252 124199 catalog_manager.cc:3791] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4f55686db84876a066f455f6a41180 (127.112.245.195). New cstate: current_term: 1 leader_uuid: "9a4f55686db84876a066f455f6a41180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: HEALTHY } } }
W0227 17:25:46.521956 125299 tablet_service.cc:1915] Rejecting scan request for tablet 9808ecd2a4be428290fa53bb51dd17d4: Uninitialized: safe time has not yet been initialized
W0227 17:25:46.725854 125299 tablet_service.cc:1915] Rejecting scan request for tablet 9808ecd2a4be428290fa53bb51dd17d4: Uninitialized: safe time has not yet been initialized
I0227 17:25:46.944555 125012 raft_consensus.cc:1181] T 9808ecd2a4be428290fa53bb51dd17d4 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:46.945300 128625 consensus_queue.cc:984] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:46.963146 124804 raft_consensus.cc:1181] T 9808ecd2a4be428290fa53bb51dd17d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:46.964776 128625 consensus_queue.cc:984] T 9808ecd2a4be428290fa53bb51dd17d4 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:47.135870 128989 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:47.135972 128989 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:47.140612 128989 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:47.141006 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:47.141080 124217 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:47.177618 124197 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.252:49654:
name: "default.table_0_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:25:47.178040 124197 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:25:47.345654 125316 tablet_service.cc:774] Processing CreateTablet for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table=default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:47.345763 124988 tablet_service.cc:774] Processing CreateTablet for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table=default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:47.345841 124778 tablet_service.cc:774] Processing CreateTablet for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table=default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:47.345894 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8c4950a6e6e54bc59d6e40c00df12fab. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:47.346012 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8c4950a6e6e54bc59d6e40c00df12fab. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:47.346065 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8c4950a6e6e54bc59d6e40c00df12fab. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:47.347774 125316 ts_tablet_manager.cc:1175] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:47.347928 124778 ts_tablet_manager.cc:1175] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:47.347930 124988 ts_tablet_manager.cc:1175] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:47.355747 129084 ts_tablet_manager.cc:1030] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:25:47.355917 129084 tablet_bootstrap.cc:439] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:25:47.356993 129084 tablet_bootstrap.cc:593] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:25:47.359000 129085 ts_tablet_manager.cc:1030] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:25:47.359151 129085 tablet_bootstrap.cc:439] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:25:47.360154 129083 ts_tablet_manager.cc:1030] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:25:47.360275 129083 tablet_bootstrap.cc:439] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:25:47.360445 129085 tablet_bootstrap.cc:593] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:25:47.362872 129083 tablet_bootstrap.cc:593] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:25:47.370787 129084 tablet_bootstrap.cc:439] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:25:47.370894 129084 ts_tablet_manager.cc:1047] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.015s	user 0.001s	sys 0.000s
I0227 17:25:47.371471 129084 raft_consensus.cc:340] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.371505 129084 raft_consensus.cc:366] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:47.371528 129084 raft_consensus.cc:705] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:25:47.371682 129084 consensus_queue.cc:229] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.371942 129084 ts_tablet_manager.cc:1075] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I0227 17:25:47.372386 129085 tablet_bootstrap.cc:439] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:25:47.372483 129085 ts_tablet_manager.cc:1047] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.000s
I0227 17:25:47.373131 129085 raft_consensus.cc:340] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.373163 129085 raft_consensus.cc:366] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:47.373186 129085 raft_consensus.cc:705] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:25:47.373332 129085 consensus_queue.cc:229] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.373442 129083 tablet_bootstrap.cc:439] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:25:47.373530 129083 ts_tablet_manager.cc:1047] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.000s
I0227 17:25:47.373587 129085 ts_tablet_manager.cc:1075] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:25:47.374068 129083 raft_consensus.cc:340] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.374092 129083 raft_consensus.cc:366] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:47.374111 129083 raft_consensus.cc:705] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:25:47.374250 129083 consensus_queue.cc:229] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.374502 129083 ts_tablet_manager.cc:1075] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0227 17:25:47.560992 129094 raft_consensus.cc:472] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:47.561198 129094 raft_consensus.cc:494] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.561717 129094 leader_election.cc:292] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:25:47.562474 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8c4950a6e6e54bc59d6e40c00df12fab" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:25:47.562577 124804 raft_consensus.cc:2346] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:25:47.563041 125257 leader_election.cc:306] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4f55686db84876a066f455f6a41180, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:25:47.563192 129094 raft_consensus.cc:2640] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:25:47.563222 129094 raft_consensus.cc:472] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:47.563238 129094 raft_consensus.cc:2886] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:47.564429 129094 raft_consensus.cc:494] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.564826 129094 leader_election.cc:292] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:25:47.565321 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8c4950a6e6e54bc59d6e40c00df12fab" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:25:47.565387 124804 raft_consensus.cc:2886] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:47.566447 125012 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8c4950a6e6e54bc59d6e40c00df12fab" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf" is_pre_election: true
I0227 17:25:47.566527 125012 raft_consensus.cc:2346] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:25:47.566597 125000 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "8c4950a6e6e54bc59d6e40c00df12fab" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf"
I0227 17:25:47.566602 124804 raft_consensus.cc:2346] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:25:47.566644 125000 raft_consensus.cc:2886] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:47.567095 125257 leader_election.cc:306] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4f55686db84876a066f455f6a41180, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:25:47.567234 129094 raft_consensus.cc:2640] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:25:47.567301 129094 raft_consensus.cc:667] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Running, Role: LEADER
I0227 17:25:47.567461 129094 consensus_queue.cc:206] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:47.567663 125000 raft_consensus.cc:2346] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:25:47.569317 124199 catalog_manager.cc:3791] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4f55686db84876a066f455f6a41180 (127.112.245.195). New cstate: current_term: 1 leader_uuid: "9a4f55686db84876a066f455f6a41180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: HEALTHY } } }
W0227 17:25:47.582511 125299 tablet_service.cc:1915] Rejecting scan request for tablet 8c4950a6e6e54bc59d6e40c00df12fab: Uninitialized: safe time has not yet been initialized
W0227 17:25:47.663321 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.522s (SENT)
I0227 17:25:47.663414 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:25:47.674139 129200 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W0227 17:25:47.784273 125299 tablet_service.cc:1915] Rejecting scan request for tablet 8c4950a6e6e54bc59d6e40c00df12fab: Uninitialized: safe time has not yet been initialized
I0227 17:25:48.017416 125000 raft_consensus.cc:1181] T 8c4950a6e6e54bc59d6e40c00df12fab P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:48.017977 129094 consensus_queue.cc:984] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:48.023000 124804 raft_consensus.cc:1181] T 8c4950a6e6e54bc59d6e40c00df12fab P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:48.023725 129094 consensus_queue.cc:984] T 8c4950a6e6e54bc59d6e40c00df12fab P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:48.188426 123877 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:48.188890 123877 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:48.195976 129528 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:48.196149 129528 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:48.196949 129528 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:48.197095 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:48.197186 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:48.495535 129691 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:48.495767 129691 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:48.496642 129691 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:48.496773 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:48.496862 123878 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
W0227 17:25:48.853969 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.654s (SENT)
I0227 17:25:48.854076 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:48.858952 130078 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:48.859118 130078 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:48.859874 130078 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:48.859997 130078 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I0227 17:25:48.860388 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:48.860510 126660 raft_consensus.cc:2293] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica has last-logged OpId of term: 3 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 12.
W0227 17:25:49.031438 126592 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.533s (SENT)
I0227 17:25:49.031525 126592 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:25:49.033428 130164 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:49.033540 130164 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:49.034015 130164 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:49.034420 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:25:49.034528 123878 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 3.
I0227 17:25:49.034847 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:25:49.037317 130164 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I0227 17:25:49.040123 130164 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Leader pre-election won for term 4
I0227 17:25:49.040163 130164 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:25:49.040196 130164 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 3 FOLLOWER]: Advancing to term 4
I0227 17:25:49.042419 130164 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:49.042940 130164 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 election: Requested vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:49.043346 123878 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418"
I0227 17:25:49.043429 123878 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 3 FOLLOWER]: Advancing to term 4
I0227 17:25:49.044644 123878 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 4.
I0227 17:25:49.044983 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:25:49.049404 130164 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 FOLLOWER]: Leader election won for term 4
I0227 17:25:49.049513 130164 raft_consensus.cc:667] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Becoming Leader. State: Replica: c31e774156b64c02922906e3e2985c89, State: Running, Role: LEADER
I0227 17:25:49.049957 130164 consensus_queue.cc:206] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:49.051746 130163 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:49.051774 130163 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:49.059542 130168 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:25:49.062024 130168 catalog_manager.cc:347] Loaded metadata for table default.table_0 [id=bc75674adf8e42de8857bcba6d648867]
I0227 17:25:49.062325 130168 catalog_manager.cc:347] Loaded metadata for table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]
I0227 17:25:49.063350 130168 catalog_manager.cc:347] Loaded metadata for table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]
I0227 17:25:49.071107 130168 catalog_manager.cc:401] Loaded metadata for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b])
I0227 17:25:49.071286 130168 catalog_manager.cc:401] Loaded metadata for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31])
I0227 17:25:49.071442 130168 catalog_manager.cc:401] Loaded metadata for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table default.table_0 [id=bc75674adf8e42de8857bcba6d648867])
I0227 17:25:49.071550 130168 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:25:49.072562 130168 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:25:49.073171 130168 catalog_manager.cc:4106] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Loaded TSK: 0
I0227 17:25:49.073807 130168 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:25:49.479619 123878 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Refusing update from remote peer c31e774156b64c02922906e3e2985c89: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I0227 17:25:49.480165 130163 consensus_queue.cc:984] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 11, Time since last communication: 0.000s
I0227 17:25:49.484690 130505 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:49.484813 130505 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:49.493042 130163 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:49.493096 130163 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:49.493767 130505 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:49.493799 130505 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:25:49.523389 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.662s (SENT)
I0227 17:25:49.523504 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:49.523710 130505 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I0227 17:25:49.628250 125384 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
W0227 17:25:49.709816 126592 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.675s (SENT)
W0227 17:25:49.835573 126592 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.792s (SENT)
I0227 17:25:50.031775 125070 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:25:50.049105 124857 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:25:50.090931 130853 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:25:50.091161 130853 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:50.092083 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:50.092160 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:50.092736 130853 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
W0227 17:25:50.907086 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.813s (SENT)
I0227 17:25:50.907169 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:50.907474 131693 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:25:51.522033 132047 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:25:51.522171 132047 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:51.522692 132047 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:51.523144 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:51.523234 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
W0227 17:25:52.121803 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.598s (SENT)
I0227 17:25:52.121879 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:52.122249 132556 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:25:52.914418 133267 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:25:52.914546 133267 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:52.915341 133267 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:52.915422 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:52.915511 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:53.525054 133627 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:25:53.525310 133627 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:53.526010 133627 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:53.526439 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:53.526525 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
W0227 17:25:53.566145 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.650s (SENT)
I0227 17:25:53.566231 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:53.566407 133627 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W0227 17:25:54.118755 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.592s (SENT)
I0227 17:25:54.118824 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:54.119055 133811 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:25:55.509497 135544 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:25:55.509640 135544 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:55.510180 135544 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:55.510635 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:55.510723 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:55.985954 124299 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 6cd5de61888340e681a3e5cd19fad418, Raft Consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:56.100972 135760 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:25:56.101110 135760 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:56.101619 135760 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:56.102051 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:56.102133 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
W0227 17:25:56.230973 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.720s (SENT)
I0227 17:25:56.231063 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:56.231225 135760 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W0227 17:25:56.759721 123721 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.657s (SENT)
I0227 17:25:56.759807 123721 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:25:56.760175 136071 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W0227 17:25:58.201686 125414 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.252:58453)
I0227 17:25:58.315160 126645 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.254:47010:
name: "default.table_0_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:25:58.316673 126645 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:25:58.412058 125316 tablet_service.cc:774] Processing CreateTablet for tablet bf54e38e762844a18884463d41a59bf2 (table=default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:58.412294 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet bf54e38e762844a18884463d41a59bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:58.412793 124778 tablet_service.cc:774] Processing CreateTablet for tablet bf54e38e762844a18884463d41a59bf2 (table=default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:58.413022 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet bf54e38e762844a18884463d41a59bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:58.413511 124988 tablet_service.cc:774] Processing CreateTablet for tablet bf54e38e762844a18884463d41a59bf2 (table=default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:25:58.413710 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet bf54e38e762844a18884463d41a59bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:25:58.414188 125316 ts_tablet_manager.cc:1175] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:58.414418 136915 ts_tablet_manager.cc:1030] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:25:58.414539 136915 tablet_bootstrap.cc:439] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:25:58.414814 124778 ts_tablet_manager.cc:1175] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:58.415029 136916 ts_tablet_manager.cc:1030] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:25:58.415118 136916 tablet_bootstrap.cc:439] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:25:58.415436 124988 ts_tablet_manager.cc:1175] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:25:58.415628 136917 ts_tablet_manager.cc:1030] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:25:58.415729 136917 tablet_bootstrap.cc:439] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:25:58.415818 136915 tablet_bootstrap.cc:593] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:25:58.416225 136916 tablet_bootstrap.cc:593] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:25:58.416566 136917 tablet_bootstrap.cc:593] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:25:58.417327 136915 tablet_bootstrap.cc:439] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:25:58.417351 136916 tablet_bootstrap.cc:439] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:25:58.417418 136916 ts_tablet_manager.cc:1047] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:25:58.417431 136915 ts_tablet_manager.cc:1047] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.001s
I0227 17:25:58.417788 136917 tablet_bootstrap.cc:439] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:25:58.417852 136917 ts_tablet_manager.cc:1047] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:25:58.418021 136916 raft_consensus.cc:340] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.418052 136916 raft_consensus.cc:366] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:58.418074 136916 raft_consensus.cc:705] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:25:58.418226 136916 consensus_queue.cc:229] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.418273 136917 raft_consensus.cc:340] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.418284 136915 raft_consensus.cc:340] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.418300 136917 raft_consensus.cc:366] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:58.418319 136917 raft_consensus.cc:705] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:25:58.418330 136915 raft_consensus.cc:366] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:25:58.418373 136915 raft_consensus.cc:705] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:25:58.418442 136917 consensus_queue.cc:229] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.418562 136916 ts_tablet_manager.cc:1075] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I0227 17:25:58.418637 136915 consensus_queue.cc:229] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.418817 136917 ts_tablet_manager.cc:1075] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:25:58.419075 136915 ts_tablet_manager.cc:1075] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:25:58.449869 128652 raft_consensus.cc:472] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:25:58.450004 128652 raft_consensus.cc:494] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.450567 128652 leader_election.cc:292] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:25:58.451128 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "bf54e38e762844a18884463d41a59bf2" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:25:58.451130 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "bf54e38e762844a18884463d41a59bf2" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:25:58.451222 125337 raft_consensus.cc:2346] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:25:58.451233 124804 raft_consensus.cc:2346] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:25:58.451642 124931 leader_election.cc:306] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9a4f55686db84876a066f455f6a41180; no voters: 
I0227 17:25:58.451802 128652 raft_consensus.cc:2640] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:25:58.451828 128652 raft_consensus.cc:472] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:25:58.451845 128652 raft_consensus.cc:2886] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:58.452854 128652 raft_consensus.cc:494] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.453234 128652 leader_election.cc:292] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Requested vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:25:58.453591 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "bf54e38e762844a18884463d41a59bf2" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:25:58.453641 124804 raft_consensus.cc:2886] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:58.453836 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "bf54e38e762844a18884463d41a59bf2" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:25:58.453877 125337 raft_consensus.cc:2886] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:25:58.454668 124804 raft_consensus.cc:2346] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:25:58.454851 125337 raft_consensus.cc:2346] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:25:58.455181 124930 leader_election.cc:306] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:25:58.455294 128652 raft_consensus.cc:2640] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:25:58.455503 128652 raft_consensus.cc:667] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [term 1 LEADER]: Becoming Leader. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Running, Role: LEADER
I0227 17:25:58.455652 128652 consensus_queue.cc:206] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:25:58.457643 126646 catalog_manager.cc:3791] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194). New cstate: current_term: 1 leader_uuid: "39144add463f4fe89dc1ea42c778d8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } }
W0227 17:25:58.468703 124966 tablet_service.cc:1915] Rejecting scan request for tablet bf54e38e762844a18884463d41a59bf2: Uninitialized: safe time has not yet been initialized
W0227 17:25:58.670326 124966 tablet_service.cc:1915] Rejecting scan request for tablet bf54e38e762844a18884463d41a59bf2: Uninitialized: safe time has not yet been initialized
I0227 17:25:58.914450 124804 raft_consensus.cc:1181] T bf54e38e762844a18884463d41a59bf2 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:58.915000 128652 consensus_queue.cc:984] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:58.918208 125337 raft_consensus.cc:1181] T bf54e38e762844a18884463d41a59bf2 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:25:58.918712 137274 consensus_queue.cc:984] T bf54e38e762844a18884463d41a59bf2 P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:25:58.962287 137294 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:25:58.962404 137294 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:58.962918 137294 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:25:58.963364 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:25:58.963446 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:59.073596 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 123549
I0227 17:25:59.073637 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.073694 124213 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.073714 124217 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:59.073776 124213 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.073977 124215 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.074118 124215 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is either leader or believes a valid leader to be alive.
I0227 17:25:59.074326 124216 raft_consensus.cc:2881] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Stepping down as leader of term 3
I0227 17:25:59.074352 124216 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2e3d7e621b864672aaa91ea2d0bbd1b9, State: Running, Role: LEADER
I0227 17:25:59.074519 124215 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.074532 124216 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:25:59.074599 124216 consensus_peers.cc:482] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 -> Peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Closing peer: 6cd5de61888340e681a3e5cd19fad418
I0227 17:25:59.074606 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.074617 124213 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
I0227 17:25:59.074599 124212 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.074599 124217 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.074736 124216 consensus_peers.cc:482] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 -> Peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Closing peer: c31e774156b64c02922906e3e2985c89
I0227 17:25:59.074865 124216 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 3 FOLLOWER]: Advancing to term 4
I0227 17:25:59.075069 124211 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.075198 124210 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.075285 126660 raft_consensus.cc:1149] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Rejecting Update request from peer 2e3d7e621b864672aaa91ea2d0bbd1b9 for earlier term 3. Current term is 4. Ops: []
I0227 17:25:59.075491 124209 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.075604 124208 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:25:59.075857 124216 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Refusing update from remote peer c31e774156b64c02922906e3e2985c89: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I0227 17:25:59.075983 124215 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076050 124213 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 4 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076123 124212 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076200 124217 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076330 124211 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076421 124210 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076488 124209 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076493 137298 consensus_queue.cc:984] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I0227 17:25:59.076545 124208 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076735 124214 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 4 because replica is already servicing an update from a current leader or another vote.
I0227 17:25:59.076990 125382 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:25:59.077376 125068 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:25:59.077666 124855 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:25:59.078791 137307 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:59.078840 137307 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:25:59.078804 124302 hms_notification_log_listener.cc:130] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events
I0227 17:25:59.079005 137305 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:59.079030 137305 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:25:59.079234 137308 mvcc.cc:203] Tried to move safe_time back from 6354195081045950464 to 6354195042511794176. Current Snapshot: MvccSnapshot[committed={T|T < 6354195081045950464}]
W0227 17:25:59.098624 124106 connection.cc:647] server connection from 127.112.245.252:49513 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0227 17:25:59.098700 124742 connection.cc:647] client connection to 127.112.245.253:50732 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.253:50732): Connection reset by peer (error 104)
W0227 17:25:59.098742 124931 connection.cc:647] client connection to 127.112.245.253:50732 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.253:50732): Connection reset by peer (error 104)
W0227 17:25:59.098763 125258 connection.cc:647] client connection to 127.112.245.253:50732 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.253:50732): Connection reset by peer (error 104)
I0227 17:25:59.219110 137313 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:59.219288 137313 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
W0227 17:25:59.221233 137313 data_dirs.cc:617] IO error: Could not lock block_manager_instance file. Make sure that Kudu is not already running and you are not trying to run Kudu with a different user than before: lock /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data/block_manager_instance: Resource temporarily unavailable (error 11)
W0227 17:25:59.221253 137313 data_dirs.cc:618] Proceeding without lock
I0227 17:25:59.225402 137313 fs_manager.cc:399] Time spent opening directory manager: real 0.005s	user 0.002s	sys 0.002s
I0227 17:25:59.225913 137313 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:59.226611 137313 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:59.229496 137313 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:59.229626 137313 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:25:59.230442 137322 log_block_manager.cc:2649] Read-only block manager, skipping repair
I0227 17:25:59.231266 137313 fs_report.cc:352] FS layout report
--------------------
wal directory: 
metadata directory: 
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:59.231328 137313 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I0227 17:25:59.231778 137313 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:25:40 on ve0518.halxg.cloudera.com"
I0227 17:25:59.246165 115671 master_failover-itest.cc:410] UUIDS: c31e774156b64c02922906e3e2985c89 6cd5de61888340e681a3e5cd19fad418 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:25:59.356539 137325 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:59.356719 137325 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
W0227 17:25:59.396376 126593 consensus_peers.cc:457] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 -> Peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Couldn't send request to peer 6cd5de61888340e681a3e5cd19fad418. Status: Network error: Client connection negotiation failed: client connection to 127.112.245.253:50732: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I0227 17:25:59.554656 137298 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:59.554697 137298 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:59.570453 137325 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/instance:
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:59 on ve0518.halxg.cloudera.com"
I0227 17:25:59.581316 137325 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal/instance:
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:59 on ve0518.halxg.cloudera.com"
I0227 17:25:59.602596 137325 fs_manager.cc:503] Time spent creating directory manager: real 0.021s	user 0.005s	sys 0.004s
I0227 17:25:59.801546 137349 fs_manager.cc:263] Metadata directory not provided
I0227 17:25:59.801775 137349 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:25:59.809697 137349 fs_manager.cc:399] Time spent opening directory manager: real 0.006s	user 0.004s	sys 0.001s
I0227 17:25:59.810729 137349 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:25:59.811719 137349 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:25:59.814558 137349 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:25:59.814674 137349 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data to 2721 blocks
I0227 17:25:59.816501 137349 fs_report.cc:352] FS layout report
--------------------
wal directory: 
metadata directory: 
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:25:59.816664 137349 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I0227 17:25:59.817329 137349 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:59 on ve0518.halxg.cloudera.com"
I0227 17:25:59.827847 137342 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:25:59.827889 137342 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:25:59.832731 137349 tablet_copy_client.cc:240] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: tablet copy: Beginning tablet copy session from remote peer at address 127.112.245.252:58453
I0227 17:25:59.850190 124222 tablet_copy_service.cc:135] P 2e3d7e621b864672aaa91ea2d0bbd1b9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6cd5de61888340e681a3e5cd19fad418 ({username='adar'} at 127.112.245.252:50299)
I0227 17:25:59.850265 124222 tablet_copy_service.cc:156] P 2e3d7e621b864672aaa91ea2d0bbd1b9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6cd5de61888340e681a3e5cd19fad418 at {username='adar'} at 127.112.245.252:50299: session id = 6cd5de61888340e681a3e5cd19fad418-00000000000000000000000000000000
I0227 17:25:59.851320 124222 tablet_copy_source_session.cc:202] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Tablet Copy: opened 0 blocks and 1 log segments
I0227 17:25:59.857396 137349 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I0227 17:25:59.870151 137349 tablet_copy_client.cc:575] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: tablet copy: Starting download of 0 data blocks...
I0227 17:25:59.871331 137349 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: tablet copy: Starting download of 1 WAL segments...
I0227 17:25:59.880936 137349 tablet_copy_client.cc:414] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I0227 17:25:59.888783 124222 tablet_copy_service.cc:325] P 2e3d7e621b864672aaa91ea2d0bbd1b9: Request end of tablet copy session 6cd5de61888340e681a3e5cd19fad418-00000000000000000000000000000000 received from {username='adar'} at 127.112.245.252:50299
I0227 17:25:59.888851 124222 tablet_copy_service.cc:413] P 2e3d7e621b864672aaa91ea2d0bbd1b9: ending tablet copy session 6cd5de61888340e681a3e5cd19fad418-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6cd5de61888340e681a3e5cd19fad418
I0227 17:25:59.907250 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:50732
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--hive_metastore_uris=thrift://127.0.0.1:54702 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:26:00.006393 137406 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:26:00.006669 137406 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:26:00.007884 137406 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:54702
W0227 17:26:00.007910 137406 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:26:00.007927 137406 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:26:00.007938 137406 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:26:00.007959 137406 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:26:00.010530 137406 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:54702
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.253:50732
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
--webserver_interface=127.112.245.253
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.137406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:26:00.020407 137406 master_main.cc:80] Initializing master server...
I0227 17:26:00.021237 137406 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 231275us
I0227 17:26:00.022807 137406 fs_manager.cc:263] Metadata directory not provided
I0227 17:26:00.022861 137406 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:26:00.028700 137406 fs_manager.cc:399] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I0227 17:26:00.029243 137406 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:26:00.029759 137406 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:26:00.032946 137406 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:26:00.033071 137406 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data to 2721 blocks
I0227 17:26:00.036178 137406 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I0227 17:26:00.036924 137406 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:59 on ve0518.halxg.cloudera.com"
I0227 17:26:00.037060 137406 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:26:00.056344 137435 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:00.056463 137435 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:00.056943 137435 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:00.057384 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:00.057471 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 5 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:00.057507 124105 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Network error: Client connection negotiation failed: client connection to 127.112.245.253:50732: connect: Connection refused (error 111)
I0227 17:26:00.057809 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:00.057926 137435 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:26:00.097865 137406 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:26:00.098759 137406 master_main.cc:83] Starting Master server...
I0227 17:26:00.105643 137406 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.253:50732
I0227 17:26:00.106071 137406 webserver.cc:175] Starting webserver on 127.112.245.253:0
I0227 17:26:00.106091 137406 webserver.cc:186] Document root disabled
I0227 17:26:00.106709 137406 webserver.cc:313] Webserver started. Bound to: http://127.112.245.253:56818/
I0227 17:26:00.107461 137406 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/info.pb
I0227 17:26:00.111148 137487 sys_catalog.cc:188] Verifying existing consensus state
I0227 17:26:00.111526 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 137406
I0227 17:26:00.112854 137487 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Bootstrap starting.
I0227 17:26:00.118758 137487 log.cc:525] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Log is configured to *not* fsync() on all Append() calls
I0227 17:26:00.120198 126651 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.254:47010:
name: "default.table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:26:00.120549 126651 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:00.134371 137487 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=13 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0227 17:26:00.134670 137487 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Bootstrap complete.
I0227 17:26:00.138664 137487 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:00.138748 137487 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Initialized, Role: FOLLOWER
I0227 17:26:00.139235 137487 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 4.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:00.139562 137487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:00.139595 137487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:00.139722 137501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:00.139767 137501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:00.140370 137487 sys_catalog.cc:424] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:26:00.141079 137503 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting
I0227 17:26:00.141371 137406 master_main.cc:86] Master server successfully started.
I0227 17:26:00.159492 137504 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: acquiring CA information for follower catalog manager: success
I0227 17:26:00.161116 137504 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:26:00.204298 124778 tablet_service.cc:774] Processing CreateTablet for tablet dd3249f630b848c8b1efd3a68065de81 (table=default.table_1 [id=7189818660584405852c5a4249b8601d]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:00.204331 125316 tablet_service.cc:774] Processing CreateTablet for tablet dd3249f630b848c8b1efd3a68065de81 (table=default.table_1 [id=7189818660584405852c5a4249b8601d]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:00.204354 124988 tablet_service.cc:774] Processing CreateTablet for tablet dd3249f630b848c8b1efd3a68065de81 (table=default.table_1 [id=7189818660584405852c5a4249b8601d]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:00.204550 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet dd3249f630b848c8b1efd3a68065de81. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:00.204550 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet dd3249f630b848c8b1efd3a68065de81. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:00.204553 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet dd3249f630b848c8b1efd3a68065de81. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:00.206277 124988 ts_tablet_manager.cc:1175] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:00.206393 124778 ts_tablet_manager.cc:1175] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:00.206611 137536 ts_tablet_manager.cc:1030] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:26:00.206665 137535 ts_tablet_manager.cc:1030] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:26:00.206748 137536 tablet_bootstrap.cc:439] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:26:00.207574 137535 tablet_bootstrap.cc:439] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:26:00.208887 137536 tablet_bootstrap.cc:593] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:26:00.208894 137535 tablet_bootstrap.cc:593] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:26:00.209547 125316 ts_tablet_manager.cc:1175] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:00.209898 137535 tablet_bootstrap.cc:439] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:26:00.210168 137536 tablet_bootstrap.cc:439] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:26:00.210176 137542 ts_tablet_manager.cc:1030] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:26:00.210238 137536 ts_tablet_manager.cc:1047] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.001s
I0227 17:26:00.210253 137535 ts_tablet_manager.cc:1047] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I0227 17:26:00.210260 137542 tablet_bootstrap.cc:439] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:26:00.210808 137536 raft_consensus.cc:340] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.210839 137536 raft_consensus.cc:366] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:00.210856 137536 raft_consensus.cc:705] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:26:00.210985 137536 consensus_queue.cc:229] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.211197 137542 tablet_bootstrap.cc:593] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:26:00.211279 137536 ts_tablet_manager.cc:1075] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I0227 17:26:00.211601 137535 raft_consensus.cc:340] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.211675 137535 raft_consensus.cc:366] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:00.211736 137535 raft_consensus.cc:705] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:26:00.212306 137535 consensus_queue.cc:229] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.212596 137535 ts_tablet_manager.cc:1075] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:26:00.213824 137542 tablet_bootstrap.cc:439] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:26:00.213955 137542 ts_tablet_manager.cc:1047] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.001s
I0227 17:26:00.214515 137542 raft_consensus.cc:340] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.214728 137542 raft_consensus.cc:366] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:00.214803 137542 raft_consensus.cc:705] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:26:00.214982 137542 consensus_queue.cc:229] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.215358 137542 ts_tablet_manager.cc:1075] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:00.220468 124856 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:00.220605 125069 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:00.221773 137453 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" instance_seqno: 1551317141335050) as {username='adar'} at 127.112.245.193:56476; Asking this server to re-register.
I0227 17:26:00.221773 137454 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" instance_seqno: 1551317141580643) as {username='adar'} at 127.112.245.194:34489; Asking this server to re-register.
I0227 17:26:00.222327 124856 heartbeater.cc:425] Registering TS with master...
I0227 17:26:00.222476 125069 heartbeater.cc:425] Registering TS with master...
I0227 17:26:00.223244 137454 ts_manager.cc:98] Registered new tserver with Master: 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:26:00.224553 137454 ts_manager.cc:98] Registered new tserver with Master: 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:00.224647 125383 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:00.225584 137454 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "9a4f55686db84876a066f455f6a41180" instance_seqno: 1551317141855821) as {username='adar'} at 127.112.245.195:51016; Asking this server to re-register.
I0227 17:26:00.226074 125383 heartbeater.cc:425] Registering TS with master...
I0227 17:26:00.226697 137454 ts_manager.cc:98] Registered new tserver with Master: 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:26:00.364411 137501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:00.364454 137501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:00.569988 129996 raft_consensus.cc:472] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:00.570155 129996 raft_consensus.cc:494] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.570742 129996 leader_election.cc:292] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:00.571410 125000 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "dd3249f630b848c8b1efd3a68065de81" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf" is_pre_election: true
I0227 17:26:00.571496 125000 raft_consensus.cc:2346] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:26:00.571501 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "dd3249f630b848c8b1efd3a68065de81" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:26:00.571596 124804 raft_consensus.cc:2346] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:26:00.571900 125257 leader_election.cc:306] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9a4f55686db84876a066f455f6a41180; no voters: 
I0227 17:26:00.572047 129996 raft_consensus.cc:2640] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:26:00.572088 129996 raft_consensus.cc:472] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:26:00.572118 129996 raft_consensus.cc:2886] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:00.573247 129996 raft_consensus.cc:494] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.573673 129996 leader_election.cc:292] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:00.574195 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "dd3249f630b848c8b1efd3a68065de81" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:26:00.574254 124804 raft_consensus.cc:2886] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:00.574826 125000 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "dd3249f630b848c8b1efd3a68065de81" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf"
I0227 17:26:00.574883 125000 raft_consensus.cc:2886] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:00.579002 125000 raft_consensus.cc:2346] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:26:00.579285 124804 raft_consensus.cc:2346] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:26:00.579495 125257 leader_election.cc:306] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9a4f55686db84876a066f455f6a41180; no voters: 
I0227 17:26:00.579694 129996 raft_consensus.cc:2640] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:26:00.579931 129996 raft_consensus.cc:667] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Running, Role: LEADER
I0227 17:26:00.580150 129996 consensus_queue.cc:206] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:00.584285 126651 catalog_manager.cc:3791] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4f55686db84876a066f455f6a41180 (127.112.245.195). New cstate: current_term: 1 leader_uuid: "9a4f55686db84876a066f455f6a41180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: UNKNOWN } } }
I0227 17:26:00.586189 137342 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:00.586228 137342 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
W0227 17:26:00.668280 125299 tablet_service.cc:1915] Rejecting scan request for tablet dd3249f630b848c8b1efd3a68065de81: Uninitialized: safe time has not yet been initialized
W0227 17:26:00.870121 125299 tablet_service.cc:1915] Rejecting scan request for tablet dd3249f630b848c8b1efd3a68065de81: Uninitialized: safe time has not yet been initialized
I0227 17:26:01.096356 138131 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:01.096583 138131 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:01.098073 138131 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:01.099139 138130 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:01.099247 138130 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:01.099721 138130 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:01.100064 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:01.100160 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:01.113400 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:01.113500 124214 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 4.
I0227 17:26:01.114079 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418; no voters: 
I0227 17:26:01.114398 138131 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Leader pre-election won for term 5
I0227 17:26:01.114446 138131 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:01.114483 138131 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 4 FOLLOWER]: Advancing to term 5
I0227 17:26:01.116575 138131 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:01.117053 138131 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 election: Requested vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:01.117404 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:01.117532 137464 raft_consensus.cc:2276] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in current term 5: Already voted for candidate 6cd5de61888340e681a3e5cd19fad418 in this term.
I0227 17:26:01.117763 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
I0227 17:26:01.117818 124214 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 4 FOLLOWER]: Advancing to term 5
I0227 17:26:01.117993 124105 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:01.118885 124214 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 5.
I0227 17:26:01.118947 138130 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I0227 17:26:01.119195 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418; no voters: 
I0227 17:26:01.119319 138131 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 FOLLOWER]: Leader election won for term 5
I0227 17:26:01.119524 138131 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Becoming Leader. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Running, Role: LEADER
I0227 17:26:01.119825 138131 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 4.24, Last appended by leader: 24, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:01.120194 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:01.120277 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:01.120708 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 5 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89"
I0227 17:26:01.120774 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Leader election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 5 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:01.125283 138139 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd5de61888340e681a3e5cd19fad418. Latest consensus state: current_term: 5 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:01.125311 138139 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:01.136561 125000 raft_consensus.cc:1181] T dd3249f630b848c8b1efd3a68065de81 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:01.137182 137834 consensus_queue.cc:984] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:01.139744 138140 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:26:01.141808 138140 catalog_manager.cc:347] Loaded metadata for table default.table_1 [id=7189818660584405852c5a4249b8601d]
I0227 17:26:01.141970 138140 catalog_manager.cc:347] Loaded metadata for table default.table_0 [id=bc75674adf8e42de8857bcba6d648867]
I0227 17:26:01.142125 138140 catalog_manager.cc:347] Loaded metadata for table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]
I0227 17:26:01.145220 138140 catalog_manager.cc:347] Loaded metadata for table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]
I0227 17:26:01.145454 138140 catalog_manager.cc:347] Loaded metadata for table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]
I0227 17:26:01.147619 124804 raft_consensus.cc:1181] T dd3249f630b848c8b1efd3a68065de81 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:01.148773 138140 catalog_manager.cc:401] Loaded metadata for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b])
I0227 17:26:01.148988 138140 catalog_manager.cc:401] Loaded metadata for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31])
I0227 17:26:01.149317 137834 consensus_queue.cc:984] T dd3249f630b848c8b1efd3a68065de81 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:01.149519 138140 catalog_manager.cc:401] Loaded metadata for tablet bf54e38e762844a18884463d41a59bf2 (table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45])
I0227 17:26:01.150197 138140 catalog_manager.cc:401] Loaded metadata for tablet dd3249f630b848c8b1efd3a68065de81 (table default.table_1 [id=7189818660584405852c5a4249b8601d])
I0227 17:26:01.152058 138140 catalog_manager.cc:401] Loaded metadata for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table default.table_0 [id=bc75674adf8e42de8857bcba6d648867])
I0227 17:26:01.152201 138140 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:26:01.153409 138140 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:26:01.154251 138140 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Loaded TSK: 0
I0227 17:26:01.157886 138140 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:26:01.190017 137464 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Rejecting Update request from peer c31e774156b64c02922906e3e2985c89 for earlier term 4. Current term is 5. Ops: []
I0227 17:26:01.190546 138158 consensus_queue.cc:995] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 }, Status: INVALID_TERM, Last received: 4.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I0227 17:26:01.190721 138158 consensus_queue.cc:1162] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Peer 6cd5de61888340e681a3e5cd19fad418 log is divergent from this leader: its last log entry 5.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I0227 17:26:01.190799 138158 raft_consensus.cc:2881] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Stepping down as leader of term 4
I0227 17:26:01.190829 138158 raft_consensus.cc:705] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c31e774156b64c02922906e3e2985c89, State: Running, Role: LEADER
I0227 17:26:01.191028 138158 consensus_queue.cc:229] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 4.24, Last appended by leader: 24, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:01.191056 138158 consensus_peers.cc:482] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 -> Peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Closing peer: 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:26:01.191095 138158 consensus_peers.cc:482] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 -> Peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Closing peer: 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:01.191236 138158 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 4 FOLLOWER]: Advancing to term 5
I0227 17:26:01.727304 138411 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:01.727532 138411 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:01.728209 138411 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:01.728704 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:01.728792 137464 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:01.730418 124214 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Refusing update from remote peer 6cd5de61888340e681a3e5cd19fad418: Log matching property violated. Preceding OpId in replica: term: 4 index: 24. Preceding OpId from leader: term: 5 index: 25. (index mismatch)
I0227 17:26:01.730933 138404 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I0227 17:26:01.731974 138411 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd5de61888340e681a3e5cd19fad418. Latest consensus state: current_term: 5 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:01.732007 138411 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:01.744453 138404 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:01.744499 138404 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:01.746326 138411 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:01.746368 138411 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:01.747601 125383 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
I0227 17:26:02.152484 125069 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
I0227 17:26:02.156651 124856 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
I0227 17:26:02.259162 138698 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:02.259388 138698 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:02.259901 138698 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:02.260776 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:02.260856 137464 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:02.651516 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.922s (SENT)
I0227 17:26:02.651602 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:02.651777 138698 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W0227 17:26:02.879974 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.619s (SENT)
I0227 17:26:02.880059 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:02.880234 138698 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I0227 17:26:03.659154 138783 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:03.659289 138783 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:03.659827 138783 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:03.660217 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:03.660290 137464 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:04.338063 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.676s (SENT)
I0227 17:26:04.338160 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:04.338440 138866 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I0227 17:26:04.989244 138919 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:04.989413 138919 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:04.990289 138919 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:04.990748 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:04.990830 137464 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:05.548573 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.557s (SENT)
I0227 17:26:05.548676 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:05.559263 138950 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I0227 17:26:07.238940 139033 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:07.239338 139033 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:07.240111 139033 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:07.240617 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:07.240698 137464 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:07.757604 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.516s (SENT)
I0227 17:26:07.757725 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:07.769222 139046 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I0227 17:26:09.627984 139320 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:09.628108 139320 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:09.628595 139320 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:09.629140 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:09.629215 137464 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:10.171253 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.541s (SENT)
I0227 17:26:10.171368 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:10.179255 139364 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I0227 17:26:11.255259 139683 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:11.255375 139683 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:11.255894 139683 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:11.256356 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:11.256433 137464 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:11.289911 125414 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.254:35995)
I0227 17:26:11.394603 137454 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.253:35523:
name: "default.table_1_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:26:11.397173 137454 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:11.512543 125316 tablet_service.cc:774] Processing CreateTablet for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table=default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:11.512639 124778 tablet_service.cc:774] Processing CreateTablet for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table=default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:11.512784 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 5c60704670c44ebf81ffea4c83d5a8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:11.512853 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 5c60704670c44ebf81ffea4c83d5a8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:11.514097 124988 tablet_service.cc:774] Processing CreateTablet for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table=default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:11.514307 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 5c60704670c44ebf81ffea4c83d5a8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:11.514678 124778 ts_tablet_manager.cc:1175] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:11.514704 125316 ts_tablet_manager.cc:1175] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:11.514912 139750 ts_tablet_manager.cc:1030] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:26:11.514930 139751 ts_tablet_manager.cc:1030] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:26:11.515035 139750 tablet_bootstrap.cc:439] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:26:11.515075 139751 tablet_bootstrap.cc:439] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:26:11.516058 124988 ts_tablet_manager.cc:1175] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:11.516103 139751 tablet_bootstrap.cc:593] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:26:11.516253 139752 ts_tablet_manager.cc:1030] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:26:11.516258 139750 tablet_bootstrap.cc:593] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:26:11.516336 139752 tablet_bootstrap.cc:439] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:26:11.517191 139752 tablet_bootstrap.cc:593] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:26:11.517412 139751 tablet_bootstrap.cc:439] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:26:11.517488 139751 ts_tablet_manager.cc:1047] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.002s	user 0.000s	sys 0.001s
I0227 17:26:11.517721 139750 tablet_bootstrap.cc:439] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:26:11.517858 139750 ts_tablet_manager.cc:1047] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I0227 17:26:11.517930 139751 raft_consensus.cc:340] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.517951 139751 raft_consensus.cc:366] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:11.517969 139751 raft_consensus.cc:705] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:26:11.518105 139751 consensus_queue.cc:229] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.518250 139752 tablet_bootstrap.cc:439] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:26:11.518317 139752 ts_tablet_manager.cc:1047] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.002s	user 0.000s	sys 0.001s
I0227 17:26:11.518328 139751 ts_tablet_manager.cc:1075] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:11.518779 139752 raft_consensus.cc:340] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.518805 139752 raft_consensus.cc:366] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:11.518822 139752 raft_consensus.cc:705] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:26:11.518927 139750 raft_consensus.cc:340] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.518954 139752 consensus_queue.cc:229] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.519135 139750 raft_consensus.cc:366] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:11.519186 139752 ts_tablet_manager.cc:1075] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I0227 17:26:11.519274 139750 raft_consensus.cc:705] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:26:11.519696 139750 consensus_queue.cc:229] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.520449 139750 ts_tablet_manager.cc:1075] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I0227 17:26:11.622689 139379 raft_consensus.cc:472] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:11.622817 139379 raft_consensus.cc:494] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.623333 139379 leader_election.cc:292] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:11.624095 124809 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "5c60704670c44ebf81ffea4c83d5a8d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:26:11.624181 124809 raft_consensus.cc:2346] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:26:11.624490 125000 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "5c60704670c44ebf81ffea4c83d5a8d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf" is_pre_election: true
I0227 17:26:11.624580 125000 raft_consensus.cc:2346] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 0.
I0227 17:26:11.624651 125257 leader_election.cc:306] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4f55686db84876a066f455f6a41180, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:11.624809 139379 raft_consensus.cc:2640] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:26:11.624837 139379 raft_consensus.cc:472] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:26:11.624867 139379 raft_consensus.cc:2886] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:11.625936 139379 raft_consensus.cc:494] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.626328 139379 leader_election.cc:292] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:11.626904 124809 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "5c60704670c44ebf81ffea4c83d5a8d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:26:11.626952 124809 raft_consensus.cc:2886] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:11.627290 125000 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "5c60704670c44ebf81ffea4c83d5a8d4" candidate_uuid: "9a4f55686db84876a066f455f6a41180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf"
I0227 17:26:11.627343 125000 raft_consensus.cc:2886] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:11.628114 124809 raft_consensus.cc:2346] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:26:11.628394 125000 raft_consensus.cc:2346] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f55686db84876a066f455f6a41180 in term 1.
I0227 17:26:11.628621 125257 leader_election.cc:306] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4f55686db84876a066f455f6a41180, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:11.628787 139379 raft_consensus.cc:2640] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:26:11.629010 139379 raft_consensus.cc:667] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Running, Role: LEADER
I0227 17:26:11.629179 139379 consensus_queue.cc:206] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:11.630887 137453 catalog_manager.cc:3791] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4f55686db84876a066f455f6a41180 (127.112.245.195). New cstate: current_term: 1 leader_uuid: "9a4f55686db84876a066f455f6a41180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: HEALTHY } } }
W0227 17:26:11.658609 125299 tablet_service.cc:1915] Rejecting scan request for tablet 5c60704670c44ebf81ffea4c83d5a8d4: Uninitialized: safe time has not yet been initialized
W0227 17:26:11.831951 124104 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.575s (SENT)
I0227 17:26:11.832093 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:11.832479 139770 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W0227 17:26:11.860286 125299 tablet_service.cc:1915] Rejecting scan request for tablet 5c60704670c44ebf81ffea4c83d5a8d4: Uninitialized: safe time has not yet been initialized
I0227 17:26:12.034621 125000 raft_consensus.cc:1181] T 5c60704670c44ebf81ffea4c83d5a8d4 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:12.035269 139379 consensus_queue.cc:984] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:12.041556 124809 raft_consensus.cc:1181] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 9a4f55686db84876a066f455f6a41180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:12.042809 139379 consensus_queue.cc:984] T 5c60704670c44ebf81ffea4c83d5a8d4 P 9a4f55686db84876a066f455f6a41180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:12.263243 138158 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:12.263343 138158 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:12.263443 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.263526 126662 raft_consensus.cc:1181] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Refusing update from remote peer 6cd5de61888340e681a3e5cd19fad418: Log matching property violated. Preceding OpId in replica: term: 4 index: 24. Preceding OpId from leader: term: 5 index: 25. (index mismatch)
I0227 17:26:12.263588 126654 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.263670 126654 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 5.
I0227 17:26:12.263726 126660 raft_consensus.cc:2321] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:12.263731 126661 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.263806 126661 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 5.
I0227 17:26:12.263808 138158 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:12.263898 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.263953 126662 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 5.
I0227 17:26:12.264053 126661 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.264123 126661 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 5.
I0227 17:26:12.264189 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.264253 126662 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 5.
I0227 17:26:12.264489 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 6 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:12.264554 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.264572 124214 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 6 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:12.264614 126662 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 5.
I0227 17:26:12.265413 125384 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:26:12.266091 124857 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:26:12.266443 125070 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:26:12.721029 140292 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:12.721323 140292 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:12.722126 140292 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:12.722601 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:12.722735 126662 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 5.
I0227 17:26:12.723156 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:26:12.723320 140292 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I0227 17:26:12.723361 140292 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:12.723410 140292 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 5 FOLLOWER]: Advancing to term 6
I0227 17:26:12.724828 140292 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:12.725399 140292 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 election: Requested vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:12.725714 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89"
I0227 17:26:12.725809 126662 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 5 FOLLOWER]: Advancing to term 6
I0227 17:26:12.727957 126662 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 6.
I0227 17:26:12.728384 124104 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:26:12.728606 140292 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 FOLLOWER]: Leader election won for term 6
I0227 17:26:12.728857 140292 raft_consensus.cc:667] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 LEADER]: Becoming Leader. State: Replica: 2e3d7e621b864672aaa91ea2d0bbd1b9, State: Running, Role: LEADER
I0227 17:26:12.729218 140292 consensus_queue.cc:206] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:12.730525 140294 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3d7e621b864672aaa91ea2d0bbd1b9. Latest consensus state: current_term: 6 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:12.730552 140294 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:12.730794 140296 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:26:12.731938 140296 catalog_manager.cc:347] Loaded metadata for table default.table_1 [id=7189818660584405852c5a4249b8601d]
I0227 17:26:12.732069 140296 catalog_manager.cc:347] Loaded metadata for table default.table_0 [id=bc75674adf8e42de8857bcba6d648867]
I0227 17:26:12.732198 140296 catalog_manager.cc:347] Loaded metadata for table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]
I0227 17:26:12.732312 140296 catalog_manager.cc:347] Loaded metadata for table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]
I0227 17:26:12.732431 140296 catalog_manager.cc:347] Loaded metadata for table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]
I0227 17:26:12.732547 140296 catalog_manager.cc:347] Loaded metadata for table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]
I0227 17:26:12.733278 140296 catalog_manager.cc:401] Loaded metadata for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276])
I0227 17:26:12.733412 140296 catalog_manager.cc:401] Loaded metadata for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b])
I0227 17:26:12.733541 140296 catalog_manager.cc:401] Loaded metadata for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31])
I0227 17:26:12.733667 140296 catalog_manager.cc:401] Loaded metadata for tablet bf54e38e762844a18884463d41a59bf2 (table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45])
I0227 17:26:12.733789 140296 catalog_manager.cc:401] Loaded metadata for tablet dd3249f630b848c8b1efd3a68065de81 (table default.table_1 [id=7189818660584405852c5a4249b8601d])
I0227 17:26:12.733912 140296 catalog_manager.cc:401] Loaded metadata for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table default.table_0 [id=bc75674adf8e42de8857bcba6d648867])
I0227 17:26:12.734038 140296 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:26:12.734854 140296 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:26:12.735421 140296 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Loaded TSK: 0
I0227 17:26:12.735641 140296 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
W0227 17:26:12.792680 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.528s (SENT)
I0227 17:26:12.792753 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:12.793036 140420 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W0227 17:26:13.234736 124105 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.512s (SENT)
I0227 17:26:13.276347 126662 raft_consensus.cc:1181] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Refusing update from remote peer 2e3d7e621b864672aaa91ea2d0bbd1b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 24. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I0227 17:26:13.276819 140639 consensus_queue.cc:984] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 24, Time since last communication: 0.000s
I0227 17:26:13.279019 140646 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:13.279062 140646 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:13.279287 140420 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3d7e621b864672aaa91ea2d0bbd1b9. Latest consensus state: current_term: 6 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:13.279314 140420 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:13.279945 140647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:13.279975 140647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:13.280444 140420 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:13.280470 140420 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:13.281395 140644 mvcc.cc:203] Tried to move safe_time back from 6354195135003103232 to 6354195091949072384. Current Snapshot: MvccSnapshot[committed={T|T < 6354195135003103232}]
W0227 17:26:13.410542 124105 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.684s (SENT)
I0227 17:26:13.545413 125068 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
I0227 17:26:13.550112 124855 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
I0227 17:26:13.611935 125382 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
I0227 17:26:13.780915 141159 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:13.781062 141159 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:13.781653 141159 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:13.782063 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:13.782151 124214 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:14.344383 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.561s (SENT)
I0227 17:26:14.344477 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:14.344800 141287 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I0227 17:26:15.120282 126738 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: c31e774156b64c02922906e3e2985c89, Raft Consensus state: current_term: 6 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:15.561985 141344 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:15.562122 141344 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:15.562820 141344 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:15.563107 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:15.563202 124214 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:16.281380 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.717s (SENT)
I0227 17:26:16.281460 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:16.290211 141539 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I0227 17:26:18.600157 143122 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:18.600412 143122 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:18.601055 143122 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:18.601567 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:18.601666 124214 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:19.110163 143378 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:19.110309 143378 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:19.110874 143378 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:19.111454 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:19.111560 124214 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:19.239596 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.637s (SENT)
I0227 17:26:19.239689 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:19.239877 143378 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W0227 17:26:19.624207 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.512s (SENT)
I0227 17:26:19.624300 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:19.624492 143378 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I0227 17:26:21.496928 146262 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:21.497081 146262 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:21.498126 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:21.498226 124214 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:21.500109 146262 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:22.119248 146951 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:22.119478 146951 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:22.121273 146951 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:22.122030 124214 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:22.122123 124214 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:22.227705 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.729s (SENT)
I0227 17:26:22.227789 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:22.228135 146951 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W0227 17:26:22.276350 125415 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.253:50732)
I0227 17:26:22.280810 124206 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.252:49654:
name: "default.table_1_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:26:22.281198 124206 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:22.511546 124778 tablet_service.cc:774] Processing CreateTablet for tablet 0156483f43f14f4e9fe0e5dc35c12333 (table=default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:22.511554 124988 tablet_service.cc:774] Processing CreateTablet for tablet 0156483f43f14f4e9fe0e5dc35c12333 (table=default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:22.511744 125316 tablet_service.cc:774] Processing CreateTablet for tablet 0156483f43f14f4e9fe0e5dc35c12333 (table=default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:22.511791 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0156483f43f14f4e9fe0e5dc35c12333. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:22.511797 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0156483f43f14f4e9fe0e5dc35c12333. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:22.511998 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0156483f43f14f4e9fe0e5dc35c12333. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:22.513662 124988 ts_tablet_manager.cc:1175] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:22.513665 124778 ts_tablet_manager.cc:1175] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:22.513975 125316 ts_tablet_manager.cc:1175] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:22.519476 147188 ts_tablet_manager.cc:1030] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:26:22.519691 147188 tablet_bootstrap.cc:439] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:26:22.520131 147189 ts_tablet_manager.cc:1030] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:26:22.520241 147189 tablet_bootstrap.cc:439] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:26:22.520968 147190 ts_tablet_manager.cc:1030] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:26:22.521121 147190 tablet_bootstrap.cc:439] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:26:22.521734 147189 tablet_bootstrap.cc:593] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:26:22.523310 147188 tablet_bootstrap.cc:593] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:26:22.524924 147190 tablet_bootstrap.cc:593] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:26:22.526381 147188 tablet_bootstrap.cc:439] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:26:22.526473 147188 ts_tablet_manager.cc:1047] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I0227 17:26:22.527000 147188 raft_consensus.cc:340] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.527024 147188 raft_consensus.cc:366] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:22.527045 147188 raft_consensus.cc:705] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:26:22.527187 147188 consensus_queue.cc:229] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.527431 147188 ts_tablet_manager.cc:1075] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:22.534610 147190 tablet_bootstrap.cc:439] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:26:22.534708 147190 ts_tablet_manager.cc:1047] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.000s
I0227 17:26:22.535245 147190 raft_consensus.cc:340] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.535270 147190 raft_consensus.cc:366] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:22.535292 147190 raft_consensus.cc:705] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:26:22.535430 147190 consensus_queue.cc:229] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.535687 147190 ts_tablet_manager.cc:1075] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0227 17:26:22.536324 147189 tablet_bootstrap.cc:439] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:26:22.536413 147189 ts_tablet_manager.cc:1047] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.016s	user 0.002s	sys 0.000s
I0227 17:26:22.536936 147189 raft_consensus.cc:340] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.536964 147189 raft_consensus.cc:366] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:22.536993 147189 raft_consensus.cc:705] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:26:22.537127 147189 consensus_queue.cc:229] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.537408 147189 ts_tablet_manager.cc:1075] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I0227 17:26:22.621515 147201 raft_consensus.cc:472] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:22.621634 147201 raft_consensus.cc:494] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.622664 147201 leader_election.cc:292] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958), 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:26:22.643128 125000 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0156483f43f14f4e9fe0e5dc35c12333" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf" is_pre_election: true
I0227 17:26:22.643227 125000 raft_consensus.cc:2346] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6459ebb80643b98e01844aa2fd6338 in term 0.
I0227 17:26:22.643705 124741 leader_election.cc:306] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:22.643990 147201 raft_consensus.cc:2640] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:26:22.644033 147201 raft_consensus.cc:472] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:26:22.644063 147201 raft_consensus.cc:2886] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:22.645258 147201 raft_consensus.cc:494] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.645691 147201 leader_election.cc:292] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 election: Requested vote from peers 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958), 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:26:22.646139 125000 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0156483f43f14f4e9fe0e5dc35c12333" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf"
I0227 17:26:22.646198 125000 raft_consensus.cc:2886] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:22.647119 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0156483f43f14f4e9fe0e5dc35c12333" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:26:22.647127 125339 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0156483f43f14f4e9fe0e5dc35c12333" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:26:22.647233 125337 raft_consensus.cc:2346] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6459ebb80643b98e01844aa2fd6338 in term 0.
I0227 17:26:22.647307 125339 raft_consensus.cc:2321] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f6459ebb80643b98e01844aa2fd6338 for term 1 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:22.647331 125000 raft_consensus.cc:2346] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6459ebb80643b98e01844aa2fd6338 in term 1.
I0227 17:26:22.647711 124741 leader_election.cc:306] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:22.648510 147201 raft_consensus.cc:2640] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:26:22.648687 147201 raft_consensus.cc:667] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 LEADER]: Becoming Leader. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Running, Role: LEADER
I0227 17:26:22.648970 147201 consensus_queue.cc:206] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:22.651087 124206 catalog_manager.cc:3791] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193). New cstate: current_term: 1 leader_uuid: "9f6459ebb80643b98e01844aa2fd6338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } }
W0227 17:26:22.664640 125071 tablet.cc:1786] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:26:22.678053 124858 tablet.cc:1786] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:26:22.692621 125385 tablet.cc:1786] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:26:22.719962 124754 tablet_service.cc:1915] Rejecting scan request for tablet 0156483f43f14f4e9fe0e5dc35c12333: Uninitialized: safe time has not yet been initialized
W0227 17:26:22.737965 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.616s (SENT)
I0227 17:26:22.738062 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:22.752358 147242 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W0227 17:26:22.922020 124754 tablet_service.cc:1915] Rejecting scan request for tablet 0156483f43f14f4e9fe0e5dc35c12333: Uninitialized: safe time has not yet been initialized
I0227 17:26:23.097388 125339 raft_consensus.cc:2886] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:23.098616 125339 raft_consensus.cc:1181] T 0156483f43f14f4e9fe0e5dc35c12333 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 9f6459ebb80643b98e01844aa2fd6338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:23.101485 147226 consensus_queue.cc:984] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:23.113158 125000 raft_consensus.cc:1181] T 0156483f43f14f4e9fe0e5dc35c12333 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Refusing update from remote peer 9f6459ebb80643b98e01844aa2fd6338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:23.113993 147201 consensus_queue.cc:984] T 0156483f43f14f4e9fe0e5dc35c12333 P 9f6459ebb80643b98e01844aa2fd6338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:23.327296 137460 raft_consensus.cc:2881] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Stepping down as leader of term 5
I0227 17:26:23.327334 137460 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Running, Role: LEADER
I0227 17:26:23.327455 137464 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.327507 137456 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.327523 137460 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:23.327554 137460 consensus_peers.cc:482] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 -> Peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Closing peer: 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:26:23.327561 137463 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 6 candidate_status { last_received { term: 4 index: 24 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.327462 137459 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.327636 137458 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.327594 137460 consensus_peers.cc:482] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 -> Peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Closing peer: c31e774156b64c02922906e3e2985c89
I0227 17:26:23.327690 137462 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418"
I0227 17:26:23.327787 137455 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.327832 137460 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 5 FOLLOWER]: Advancing to term 6
I0227 17:26:23.328986 137460 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Refusing update from remote peer 2e3d7e621b864672aaa91ea2d0bbd1b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I0227 17:26:23.329165 124856 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:23.329753 137464 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:23.329828 137456 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:23.329890 137463 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 6 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:23.329951 137459 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:23.330014 137458 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:23.330104 137462 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 6 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:23.330307 137455 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 7 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:23.330773 137503 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 6cd5de61888340e681a3e5cd19fad418, Raft Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:23.330905 137453 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.253:35523:
name: "default.table_1_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:26:23.331251 137453 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:23.331641 125383 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:23.331647 137457 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.331745 137457 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 6.
I0227 17:26:23.331919 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.331987 137461 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 6.
I0227 17:26:23.332463 125069 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:23.588863  1008 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:23.589030  1008 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:23.590102  1008 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:23.590466 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:23.590575 137461 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 6.
I0227 17:26:23.590904 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:26:23.591086  1008 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Leader pre-election won for term 7
I0227 17:26:23.591123  1008 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:23.591158  1008 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 6 FOLLOWER]: Advancing to term 7
I0227 17:26:23.592367  1008 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:23.593044  1008 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 election: Requested vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:23.593132 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418"
I0227 17:26:23.593199 137461 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 6 FOLLOWER]: Advancing to term 7
I0227 17:26:23.594444 137461 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 7.
I0227 17:26:23.594748 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:26:23.594913  1008 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 FOLLOWER]: Leader election won for term 7
I0227 17:26:23.595070  1008 raft_consensus.cc:667] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Becoming Leader. State: Replica: c31e774156b64c02922906e3e2985c89, State: Running, Role: LEADER
I0227 17:26:23.595257  1008 consensus_queue.cc:206] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 34, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:23.596047  1008 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 7 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:23.596148  1008 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:23.602555  1020 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:26:23.603984  1020 catalog_manager.cc:347] Loaded metadata for table default.table_1 [id=7189818660584405852c5a4249b8601d]
I0227 17:26:23.604146  1020 catalog_manager.cc:347] Loaded metadata for table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f]
I0227 17:26:23.604288  1020 catalog_manager.cc:347] Loaded metadata for table default.table_0 [id=bc75674adf8e42de8857bcba6d648867]
I0227 17:26:23.604429  1020 catalog_manager.cc:347] Loaded metadata for table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]
I0227 17:26:23.604563  1020 catalog_manager.cc:347] Loaded metadata for table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]
I0227 17:26:23.604706  1020 catalog_manager.cc:347] Loaded metadata for table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]
I0227 17:26:23.604838  1020 catalog_manager.cc:347] Loaded metadata for table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]
I0227 17:26:23.605664  1020 catalog_manager.cc:401] Loaded metadata for tablet 0156483f43f14f4e9fe0e5dc35c12333 (table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f])
I0227 17:26:23.605842  1020 catalog_manager.cc:401] Loaded metadata for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276])
I0227 17:26:23.605998  1020 catalog_manager.cc:401] Loaded metadata for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b])
I0227 17:26:23.611500  1020 catalog_manager.cc:401] Loaded metadata for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31])
I0227 17:26:23.611666  1020 catalog_manager.cc:401] Loaded metadata for tablet bf54e38e762844a18884463d41a59bf2 (table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45])
I0227 17:26:23.611829  1020 catalog_manager.cc:401] Loaded metadata for tablet dd3249f630b848c8b1efd3a68065de81 (table default.table_1 [id=7189818660584405852c5a4249b8601d])
I0227 17:26:23.611984  1020 catalog_manager.cc:401] Loaded metadata for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table default.table_0 [id=bc75674adf8e42de8857bcba6d648867])
I0227 17:26:23.612212  1020 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:26:23.613273  1020 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:26:23.620066  1020 catalog_manager.cc:4106] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Loaded TSK: 0
I0227 17:26:23.620448  1020 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
17:26:23.694 [HMS - ERROR - pool-8-thread-3] (RetryingHMSHandler.java:204) AlreadyExistsException(message:Table table_1_1 already exists)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1452)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1539)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
	at com.sun.proxy.$Proxy24.create_table_with_environment_context(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11191)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11175)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

W0227 17:26:23.701603 137453 catalog_manager.cc:1615] Already present: an error occurred while creating table default.table_1_1 in the HMS: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table table_1_1 already exists)
I0227 17:26:23.981508 137461 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Refusing update from remote peer c31e774156b64c02922906e3e2985c89: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I0227 17:26:23.982065  1019 consensus_queue.cc:984] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 29, Time since last communication: 0.000s
I0227 17:26:23.986063  1176 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 7 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:23.986104  1176 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:23.986212  1176 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:23.986235  1176 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:24.000869  1019 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:24.000916  1019 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:24.001260  1179 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:24.001296  1179 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:24.019680  1177 mvcc.cc:203] Tried to move safe_time back from 6354195180141555712 to 6354195139502178304. Current Snapshot: MvccSnapshot[committed={T|T < 6354195180141555712}]
W0227 17:26:24.196314 126592 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.605s (SENT)
W0227 17:26:24.322721 126592 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.726s (SENT)
I0227 17:26:24.510143  1381 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:24.510277  1381 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:24.510824  1381 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:24.511296 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 35 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:24.511397 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:24.518443 124857 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:26:24.622576 125384 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:26:24.624105 125070 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
W0227 17:26:25.012055 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.500s (SENT)
I0227 17:26:25.012136 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:25.022271  1593 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I0227 17:26:25.609335  2100 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:25.611374  2100 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:25.612025  2100 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:25.612525 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 35 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:25.612622 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:26.147892 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.534s (SENT)
I0227 17:26:26.147976 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:26.155721  2851 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I0227 17:26:28.130208  5621 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:28.130398  5621 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:28.130993  5621 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:28.131470 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 35 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:28.131573 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:28.649019 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.517s (SENT)
I0227 17:26:28.649129 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:28.659353  6394 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I0227 17:26:29.339280  7096 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:29.341259  7096 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:29.341969  7096 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:29.342483 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 35 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:29.342582 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:29.888691 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.545s (SENT)
I0227 17:26:29.888795 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:29.900154  7487 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I0227 17:26:31.502770  9018 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:31.503281  9018 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:31.503908  9018 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:31.504523 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 36 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:31.504613 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:32.030171  9546 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:32.030341  9546 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:32.030964  9546 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:32.031411 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 36 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:32.031494 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:32.065444 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.560s (SENT)
I0227 17:26:32.065526 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:32.065701  9546 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W0227 17:26:32.734139 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.700s (SENT)
I0227 17:26:32.734210 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:32.737154 10076 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W0227 17:26:33.340632 125414 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.252:58453)
I0227 17:26:33.345978 126649 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.254:47010:
name: "default.table_1_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:26:33.346349 126649 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:33.515275 124988 tablet_service.cc:774] Processing CreateTablet for tablet 7fae960e7b5343338ea18dbb27e5f63a (table=default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:33.515287 125316 tablet_service.cc:774] Processing CreateTablet for tablet 7fae960e7b5343338ea18dbb27e5f63a (table=default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:33.515514 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7fae960e7b5343338ea18dbb27e5f63a. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:33.515527 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7fae960e7b5343338ea18dbb27e5f63a. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:33.517374 124988 ts_tablet_manager.cc:1175] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:33.517395 124778 tablet_service.cc:774] Processing CreateTablet for tablet 7fae960e7b5343338ea18dbb27e5f63a (table=default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:33.517499 125316 ts_tablet_manager.cc:1175] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:33.517619 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 7fae960e7b5343338ea18dbb27e5f63a. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:33.519513 124778 ts_tablet_manager.cc:1175] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:33.522480 10499 ts_tablet_manager.cc:1030] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:26:33.522640 10499 tablet_bootstrap.cc:439] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:26:33.522750 10498 ts_tablet_manager.cc:1030] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:26:33.522929 10498 tablet_bootstrap.cc:439] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:26:33.523975 10500 ts_tablet_manager.cc:1030] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:26:33.524292 10500 tablet_bootstrap.cc:439] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:26:33.525262 10500 tablet_bootstrap.cc:593] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:26:33.525436 10498 tablet_bootstrap.cc:593] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:26:33.530966 10499 tablet_bootstrap.cc:593] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:26:33.535265 10499 tablet_bootstrap.cc:439] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:26:33.535362 10499 ts_tablet_manager.cc:1047] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.001s
I0227 17:26:33.535722 10498 tablet_bootstrap.cc:439] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:26:33.535820 10498 ts_tablet_manager.cc:1047] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.001s
I0227 17:26:33.535926 10499 raft_consensus.cc:340] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.535948 10499 raft_consensus.cc:366] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:33.535986 10499 raft_consensus.cc:705] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:26:33.536128 10499 consensus_queue.cc:229] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.536360 10499 ts_tablet_manager.cc:1075] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0227 17:26:33.536393 10498 raft_consensus.cc:340] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.536417 10498 raft_consensus.cc:366] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:33.536438 10498 raft_consensus.cc:705] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:26:33.536576 10498 consensus_queue.cc:229] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.536818 10498 ts_tablet_manager.cc:1075] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0227 17:26:33.545306 10500 tablet_bootstrap.cc:439] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:26:33.545406 10500 ts_tablet_manager.cc:1047] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.021s	user 0.001s	sys 0.000s
I0227 17:26:33.545989 10500 raft_consensus.cc:340] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.546021 10500 raft_consensus.cc:366] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:33.546044 10500 raft_consensus.cc:705] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:26:33.546190 10500 consensus_queue.cc:229] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.546463 10500 ts_tablet_manager.cc:1075] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:33.616132  9007 raft_consensus.cc:472] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:33.616267  9007 raft_consensus.cc:494] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.616798  9007 leader_election.cc:292] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:33.617445 125012 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "7fae960e7b5343338ea18dbb27e5f63a" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf" is_pre_election: true
I0227 17:26:33.617539 125012 raft_consensus.cc:2346] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6459ebb80643b98e01844aa2fd6338 in term 0.
I0227 17:26:33.617584 125339 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "7fae960e7b5343338ea18dbb27e5f63a" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:26:33.617677 125339 raft_consensus.cc:2346] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6459ebb80643b98e01844aa2fd6338 in term 0.
I0227 17:26:33.618106 124741 leader_election.cc:306] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:33.618271  9007 raft_consensus.cc:2640] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:26:33.618297  9007 raft_consensus.cc:472] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:26:33.618322  9007 raft_consensus.cc:2886] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:33.619525  9007 raft_consensus.cc:494] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.619953  9007 leader_election.cc:292] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:33.620456 125012 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "7fae960e7b5343338ea18dbb27e5f63a" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39144add463f4fe89dc1ea42c778d8cf"
I0227 17:26:33.620513 125012 raft_consensus.cc:2886] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:33.620621 125339 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "7fae960e7b5343338ea18dbb27e5f63a" candidate_uuid: "9f6459ebb80643b98e01844aa2fd6338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:26:33.620687 125339 raft_consensus.cc:2886] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:33.621599 125012 raft_consensus.cc:2346] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6459ebb80643b98e01844aa2fd6338 in term 1.
I0227 17:26:33.621803 125339 raft_consensus.cc:2346] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6459ebb80643b98e01844aa2fd6338 in term 1.
I0227 17:26:33.622025 124741 leader_election.cc:306] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:33.622288  9007 raft_consensus.cc:2640] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:26:33.622463  9007 raft_consensus.cc:667] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [term 1 LEADER]: Becoming Leader. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Running, Role: LEADER
I0227 17:26:33.622633  9007 consensus_queue.cc:206] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:33.624747 126649 catalog_manager.cc:3791] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193). New cstate: current_term: 1 leader_uuid: "9f6459ebb80643b98e01844aa2fd6338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: UNKNOWN } } }
W0227 17:26:33.643440 124754 tablet_service.cc:1915] Rejecting scan request for tablet 7fae960e7b5343338ea18dbb27e5f63a: Uninitialized: safe time has not yet been initialized
W0227 17:26:33.845377 124754 tablet_service.cc:1915] Rejecting scan request for tablet 7fae960e7b5343338ea18dbb27e5f63a: Uninitialized: safe time has not yet been initialized
I0227 17:26:34.098402 125012 raft_consensus.cc:1181] T 7fae960e7b5343338ea18dbb27e5f63a P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Refusing update from remote peer 9f6459ebb80643b98e01844aa2fd6338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:34.099112 10568 consensus_queue.cc:984] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:34.112283 125339 raft_consensus.cc:1181] T 7fae960e7b5343338ea18dbb27e5f63a P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 9f6459ebb80643b98e01844aa2fd6338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:34.113469  9007 consensus_queue.cc:984] T 7fae960e7b5343338ea18dbb27e5f63a P 9f6459ebb80643b98e01844aa2fd6338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:34.129050 10990 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:34.130528 10990 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:34.133713 10990 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:34.134191 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 39 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:34.134284 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:34.248967 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 123923
W0227 17:26:34.264843 137432 connection.cc:647] client connection to 127.112.245.252:58453 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0227 17:26:34.264927 126592 connection.cc:647] client connection to 127.112.245.252:58453 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0227 17:26:34.265014 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W0227 17:26:34.265054 126592 consensus_peers.cc:457] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 -> Peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Couldn't send request to peer 2e3d7e621b864672aaa91ea2d0bbd1b9. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W0227 17:26:34.265092 124741 connection.cc:647] client connection to 127.112.245.252:58453 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
I0227 17:26:34.265100 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:34.265326 10990 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W0227 17:26:34.265364 124741 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0227 17:26:34.265563 124930 connection.cc:647] client connection to 127.112.245.252:58453 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0227 17:26:34.265632 125257 connection.cc:647] client connection to 127.112.245.252:58453 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0227 17:26:34.265803 124930 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0227 17:26:34.265895 125257 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0227 17:26:34.271654 137430 connection.cc:647] server connection from 127.112.245.253:35450 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
I0227 17:26:34.529531 11119 fs_manager.cc:263] Metadata directory not provided
I0227 17:26:34.529829 11119 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
W0227 17:26:34.532821 11119 data_dirs.cc:617] IO error: Could not lock block_manager_instance file. Make sure that Kudu is not already running and you are not trying to run Kudu with a different user than before: lock /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data/block_manager_instance: Resource temporarily unavailable (error 11)
W0227 17:26:34.532948 11119 data_dirs.cc:618] Proceeding without lock
I0227 17:26:34.551347 11119 fs_manager.cc:399] Time spent opening directory manager: real 0.020s	user 0.006s	sys 0.000s
I0227 17:26:34.552088 11119 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:26:34.552907 11119 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:26:34.556313 11119 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:26:34.556505 11119 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data to 2721 blocks
I0227 17:26:34.557524 11528 log_block_manager.cc:2649] Read-only block manager, skipping repair
I0227 17:26:34.558431 11119 fs_report.cc:352] FS layout report
--------------------
wal directory: 
metadata directory: 
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:26:34.558614 11119 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I0227 17:26:34.559144 11119 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-1/wal
uuid: "6cd5de61888340e681a3e5cd19fad418"
format_stamp: "Formatted at 2019-02-28 01:25:59 on ve0518.halxg.cloudera.com"
I0227 17:26:34.580358 115671 master_failover-itest.cc:410] UUIDS: c31e774156b64c02922906e3e2985c89 6cd5de61888340e681a3e5cd19fad418 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:26:34.842216 11550 fs_manager.cc:263] Metadata directory not provided
I0227 17:26:34.852756 11550 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:26:34.874415 11550 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/instance:
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:26:34 on ve0518.halxg.cloudera.com"
I0227 17:26:34.885232 11550 fs_manager.cc:602] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal/instance:
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:26:34 on ve0518.halxg.cloudera.com"
I0227 17:26:34.927845 11550 fs_manager.cc:503] Time spent creating directory manager: real 0.042s	user 0.010s	sys 0.001s
I0227 17:26:35.150192 12240 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:35.150446 12240 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:35.151124 12240 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:35.151965 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 39 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:35.152070 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:35.159162 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
I0227 17:26:35.159248 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:35.159446 12240 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I0227 17:26:35.183254 12033 fs_manager.cc:263] Metadata directory not provided
I0227 17:26:35.183601 12033 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:26:35.204895 12033 fs_manager.cc:399] Time spent opening directory manager: real 0.019s	user 0.002s	sys 0.003s
I0227 17:26:35.205865 12033 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:26:35.206715 12033 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:26:35.209683 12033 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:26:35.209849 12033 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:26:35.213428 12033 fs_report.cc:352] FS layout report
--------------------
wal directory: 
metadata directory: 
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:26:35.213620 12033 fs_manager.cc:419] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.001s
I0227 17:26:35.214182 12033 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:26:34 on ve0518.halxg.cloudera.com"
I0227 17:26:35.234952 12033 tablet_copy_client.cc:240] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: tablet copy: Beginning tablet copy session from remote peer at address 127.112.245.253:50732
I0227 17:26:35.279134 137474 tablet_copy_service.cc:135] P 6cd5de61888340e681a3e5cd19fad418: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2e3d7e621b864672aaa91ea2d0bbd1b9 ({username='adar'} at 127.112.245.253:36320)
I0227 17:26:35.279219 137474 tablet_copy_service.cc:156] P 6cd5de61888340e681a3e5cd19fad418: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2e3d7e621b864672aaa91ea2d0bbd1b9 at {username='adar'} at 127.112.245.253:36320: session id = 2e3d7e621b864672aaa91ea2d0bbd1b9-00000000000000000000000000000000
I0227 17:26:35.280452 137474 tablet_copy_source_session.cc:202] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Tablet Copy: opened 0 blocks and 1 log segments
I0227 17:26:35.284880 12033 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I0227 17:26:35.404434 12033 tablet_copy_client.cc:575] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: tablet copy: Starting download of 0 data blocks...
I0227 17:26:35.411152 12033 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: tablet copy: Starting download of 1 WAL segments...
I0227 17:26:35.414502 12033 tablet_copy_client.cc:414] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I0227 17:26:35.418594 137474 tablet_copy_service.cc:325] P 6cd5de61888340e681a3e5cd19fad418: Request end of tablet copy session 2e3d7e621b864672aaa91ea2d0bbd1b9-00000000000000000000000000000000 received from {username='adar'} at 127.112.245.253:36320
I0227 17:26:35.418651 137474 tablet_copy_service.cc:413] P 6cd5de61888340e681a3e5cd19fad418: ending tablet copy session 2e3d7e621b864672aaa91ea2d0bbd1b9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:26:35.434808 115671 external_mini_cluster.cc:933] Running /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
/data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:58453
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--leader_failure_max_missed_heartbeat_periods=1.0
--rpc_reuseport=true
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--hive_metastore_uris=thrift://127.0.0.1:54702 with env {}
I0227 17:26:35.712608 13155 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:35.712743 13155 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:35.713666 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 39 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:35.713757 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:35.714473 13155 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
W0227 17:26:35.719501 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Network error: Client connection negotiation failed: client connection to 127.112.245.252:58453: connect: Connection refused (error 111)
I0227 17:26:35.719580 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:35.719863 13155 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0227 17:26:35.768788 12784 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W0227 17:26:35.769078 12784 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W0227 17:26:35.770489 12784 flags.cc:406] Enabled experimental flag: --hive_metastore_uris=thrift://127.0.0.1:54702
W0227 17:26:35.770717 12784 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W0227 17:26:35.770836 12784 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W0227 17:26:35.770890 12784 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W0227 17:26:35.770941 12784 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I0227 17:26:35.773941 12784 master_main.cc:73] Master server non-default flags:
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data
--fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:54702
--ipki_ca_key_size=1024
--master_addresses=127.112.245.254:35995,127.112.245.253:50732,127.112.245.252:58453
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.112.245.252:58453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
--webserver_interface=127.112.245.252
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu-master.12784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0
revision non-git-build
build type DEBUG
built by adar at 27 Feb 2019 17:20:38 PST on ve0518.halxg.cloudera.com
I0227 17:26:35.784598 12784 master_main.cc:80] Initializing master server...
I0227 17:26:35.786067 12784 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 248775us
I0227 17:26:35.787616 12784 fs_manager.cc:263] Metadata directory not provided
I0227 17:26:35.787729 12784 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I0227 17:26:35.800107 12784 fs_manager.cc:399] Time spent opening directory manager: real 0.010s	user 0.004s	sys 0.000s
I0227 17:26:35.802166 12784 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I0227 17:26:35.803119 12784 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I0227 17:26:35.808048 12784 log_block_manager.cc:1950] Data dir /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0227 17:26:35.808279 12784 log_block_manager.cc:1960] Limiting containers on data directory /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data to 2721 blocks
I0227 17:26:35.821375 12784 fs_manager.cc:419] Time spent opening block manager: real 0.015s	user 0.001s	sys 0.000s
I0227 17:26:35.824203 12784 fs_manager.cc:436] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data,/tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
format_stamp: "Formatted at 2019-02-28 01:26:34 on ve0518.halxg.cloudera.com"
I0227 17:26:35.824412 12784 fs_report.cc:352] FS layout report
--------------------
wal directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
metadata directory: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/wal
1 data directories: /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0227 17:26:35.897506 12784 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I0227 17:26:35.898519 12784 master_main.cc:83] Starting Master server...
I0227 17:26:35.904738 12784 rpc_server.cc:205] RPC server started. Bound to: 127.112.245.252:58453
I0227 17:26:35.916587 12784 webserver.cc:175] Starting webserver on 127.112.245.252:0
I0227 17:26:35.916725 12784 webserver.cc:186] Document root disabled
I0227 17:26:35.917511 12784 webserver.cc:313] Webserver started. Bound to: http://127.112.245.252:41023/
I0227 17:26:35.918390 12784 server_base.cc:607] Dumped server information to /tmp/kudutest-4282/master_failover-itest.1.HmsConfigurations_MasterFailoverTest.TestMasterPermanentFailure_1.1551317010817988-115671/minicluster-data/master-2/data/info.pb
I0227 17:26:35.924890 115671 external_mini_cluster.cc:995] Started /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master as pid 12784
I0227 17:26:35.936524 13646 sys_catalog.cc:188] Verifying existing consensus state
I0227 17:26:35.941053 13646 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Bootstrap starting.
I0227 17:26:35.961371 126644 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.254:47010:
name: "default.table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0227 17:26:35.961503 13646 log.cc:525] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Log is configured to *not* fsync() on all Append() calls
W0227 17:26:35.961764 126644 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:36.031599 13646 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=39 ignored=0} inserts{seen=25 ignored=0} mutations{seen=16 ignored=0} orphaned_commits=0. Pending: 0 replicates
I0227 17:26:36.032083 13646 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Bootstrap complete.
I0227 17:26:36.041451 13646 raft_consensus.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:36.043334 13646 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3d7e621b864672aaa91ea2d0bbd1b9, State: Initialized, Role: FOLLOWER
I0227 17:26:36.050477 13646 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 7.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:36.051048 13646 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:36.051147 13646 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:36.052417 13646 sys_catalog.cc:424] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: configured and running, proceeding with master startup.
I0227 17:26:36.053762 13929 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:36.054003 12784 master_main.cc:86] Master server successfully started.
I0227 17:26:36.054023 13929 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:36.054533 13944 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: 2e3d7e621b864672aaa91ea2d0bbd1b9, Raft Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:36.064546 13945 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: acquiring CA information for follower catalog manager: success
I0227 17:26:36.068701 13945 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I0227 17:26:36.074683 125316 tablet_service.cc:774] Processing CreateTablet for tablet 363dd021fd184929bc8b1f461f8fdb16 (table=default.table_2 [id=1386cdb42e004032916576ef12b3f5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:36.074841 124778 tablet_service.cc:774] Processing CreateTablet for tablet 363dd021fd184929bc8b1f461f8fdb16 (table=default.table_2 [id=1386cdb42e004032916576ef12b3f5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:36.074950 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 363dd021fd184929bc8b1f461f8fdb16. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:36.075085 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 363dd021fd184929bc8b1f461f8fdb16. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:36.075125 124988 tablet_service.cc:774] Processing CreateTablet for tablet 363dd021fd184929bc8b1f461f8fdb16 (table=default.table_2 [id=1386cdb42e004032916576ef12b3f5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:36.075325 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 363dd021fd184929bc8b1f461f8fdb16. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:36.076911 125316 ts_tablet_manager.cc:1175] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:36.076915 124778 ts_tablet_manager.cc:1175] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:36.077117 124988 ts_tablet_manager.cc:1175] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:36.083578 13987 ts_tablet_manager.cc:1030] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:26:36.083762 13987 tablet_bootstrap.cc:439] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:26:36.084959 13988 ts_tablet_manager.cc:1030] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:26:36.085362 13987 tablet_bootstrap.cc:593] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:26:36.085517 13988 tablet_bootstrap.cc:439] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:26:36.086575 13988 tablet_bootstrap.cc:593] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:26:36.086695 13992 ts_tablet_manager.cc:1030] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:26:36.086885 13992 tablet_bootstrap.cc:439] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:26:36.088299 13992 tablet_bootstrap.cc:593] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:26:36.091913 13992 tablet_bootstrap.cc:439] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:26:36.092101 13992 ts_tablet_manager.cc:1047] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.002s
I0227 17:26:36.092706 13992 raft_consensus.cc:340] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.092797 13992 raft_consensus.cc:366] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:36.092880 13992 raft_consensus.cc:705] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:26:36.093140 13992 consensus_queue.cc:229] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.093585 13992 ts_tablet_manager.cc:1075] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:36.095401 13987 tablet_bootstrap.cc:439] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:26:36.095510 13987 ts_tablet_manager.cc:1047] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.000s
I0227 17:26:36.096119 13987 raft_consensus.cc:340] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.096127 13988 tablet_bootstrap.cc:439] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:26:36.096153 13987 raft_consensus.cc:366] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:36.096176 13987 raft_consensus.cc:705] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:26:36.096233 13988 ts_tablet_manager.cc:1047] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.000s
I0227 17:26:36.096338 13987 consensus_queue.cc:229] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.096593 13987 ts_tablet_manager.cc:1075] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I0227 17:26:36.096897 13988 raft_consensus.cc:340] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.096932 13988 raft_consensus.cc:366] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:36.096956 13988 raft_consensus.cc:705] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:26:36.097124 13988 consensus_queue.cc:229] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.097410 13988 ts_tablet_manager.cc:1075] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:36.112632 124855 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:26:36.113065 125068 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:26:36.114789 13602 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" instance_seqno: 1551317141580643) as {username='adar'} at 127.112.245.194:50217; Asking this server to re-register.
I0227 17:26:36.114939 13597 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" instance_seqno: 1551317141335050) as {username='adar'} at 127.112.245.193:47215; Asking this server to re-register.
I0227 17:26:36.115324 125068 heartbeater.cc:425] Registering TS with master...
I0227 17:26:36.115422 124855 heartbeater.cc:425] Registering TS with master...
I0227 17:26:36.116309 13597 ts_manager.cc:98] Registered new tserver with Master: 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194:55958)
I0227 17:26:36.117444 13597 ts_manager.cc:98] Registered new tserver with Master: 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:26:36.122052 125382 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:26:36.123447 13597 master_service.cc:219] Got heartbeat from unknown tserver (permanent_uuid: "9a4f55686db84876a066f455f6a41180" instance_seqno: 1551317141855821) as {username='adar'} at 127.112.245.195:48937; Asking this server to re-register.
I0227 17:26:36.124002 125382 heartbeater.cc:425] Registering TS with master...
I0227 17:26:36.124855 13597 ts_manager.cc:98] Registered new tserver with Master: 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:26:36.179955  1107 raft_consensus.cc:472] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:36.180096  1107 raft_consensus.cc:494] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.180629  1107 leader_election.cc:292] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:26:36.181188 125339 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "363dd021fd184929bc8b1f461f8fdb16" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:26:36.181285 125339 raft_consensus.cc:2346] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:26:36.181285 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "363dd021fd184929bc8b1f461f8fdb16" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:26:36.181387 124804 raft_consensus.cc:2346] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:26:36.181824 124931 leader_election.cc:306] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9a4f55686db84876a066f455f6a41180; no voters: 
I0227 17:26:36.181995  1107 raft_consensus.cc:2640] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:26:36.182024  1107 raft_consensus.cc:472] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:26:36.182049  1107 raft_consensus.cc:2886] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:36.183259  1107 raft_consensus.cc:494] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.183709  1107 leader_election.cc:292] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Requested vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:26:36.184201 125339 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "363dd021fd184929bc8b1f461f8fdb16" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:26:36.184242 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "363dd021fd184929bc8b1f461f8fdb16" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:26:36.184265 125339 raft_consensus.cc:2886] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:36.184299 124804 raft_consensus.cc:2886] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:36.185230 125339 raft_consensus.cc:2346] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:26:36.185405 124804 raft_consensus.cc:2346] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:26:36.185612 124931 leader_election.cc:306] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9a4f55686db84876a066f455f6a41180; no voters: 
I0227 17:26:36.185763  1107 raft_consensus.cc:2640] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:26:36.185991  1107 raft_consensus.cc:667] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [term 1 LEADER]: Becoming Leader. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Running, Role: LEADER
I0227 17:26:36.186194  1107 consensus_queue.cc:206] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:26:36.190215 126644 catalog_manager.cc:3791] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194). New cstate: current_term: 1 leader_uuid: "39144add463f4fe89dc1ea42c778d8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: HEALTHY } } }
W0227 17:26:36.203832 124966 tablet_service.cc:1915] Rejecting scan request for tablet 363dd021fd184929bc8b1f461f8fdb16: Uninitialized: safe time has not yet been initialized
I0227 17:26:36.284778 13605 raft_consensus.cc:1126] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 6.34->[7.35-7.43]   Dedup: 7.39->[7.40-7.43]
I0227 17:26:36.288486 13929 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 7 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:36.288525 13929 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:26:36.405570 124966 tablet_service.cc:1915] Rejecting scan request for tablet 363dd021fd184929bc8b1f461f8fdb16: Uninitialized: safe time has not yet been initialized
I0227 17:26:36.618199 125339 raft_consensus.cc:1181] T 363dd021fd184929bc8b1f461f8fdb16 P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:36.618842 14264 consensus_queue.cc:984] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:36.622671 124804 raft_consensus.cc:1181] T 363dd021fd184929bc8b1f461f8fdb16 P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:36.623266  1107 consensus_queue.cc:984] T 363dd021fd184929bc8b1f461f8fdb16 P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:36.696044 14846 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:36.696166 14846 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:36.696848 14846 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:36.697574 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:36.697659 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:36.701766 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:36.701949 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:36.702419 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:26:36.702566 14846 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I0227 17:26:36.742488 13662 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:36.742527 13662 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:37.243793 15997 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:37.243961 15997 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:37.245343 15997 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:37.258069 15998 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:37.258183 15998 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:37.258620 15998 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:37.259065 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:37.259193 13605 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 7.
I0227 17:26:37.259546 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418; no voters: 
I0227 17:26:37.260082 15998 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Leader pre-election won for term 8
I0227 17:26:37.260108 15998 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:37.260125 15998 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 7 FOLLOWER]: Advancing to term 8
I0227 17:26:37.261227 15998 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:37.261603 15998 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 election: Requested vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:37.261972 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
I0227 17:26:37.262043 13605 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 7 FOLLOWER]: Advancing to term 8
I0227 17:26:37.262189 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:37.262248 137461 raft_consensus.cc:2276] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in current term 8: Already voted for candidate 6cd5de61888340e681a3e5cd19fad418 in this term.
I0227 17:26:37.264392 13605 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cd5de61888340e681a3e5cd19fad418 in term 8.
I0227 17:26:37.264855 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418; no voters: 
I0227 17:26:37.264989 15998 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 FOLLOWER]: Leader election won for term 8
I0227 17:26:37.265172 15998 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Becoming Leader. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Running, Role: LEADER
I0227 17:26:37.265348 15998 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:37.267307 16017 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd5de61888340e681a3e5cd19fad418. Latest consensus state: current_term: 8 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:37.267333 16017 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:37.267546 16019 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:26:37.271445 16019 catalog_manager.cc:347] Loaded metadata for table default.table_2 [id=1386cdb42e004032916576ef12b3f5cd]
I0227 17:26:37.271733 16019 catalog_manager.cc:347] Loaded metadata for table default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff]
I0227 17:26:37.271893 16019 catalog_manager.cc:347] Loaded metadata for table default.table_1 [id=7189818660584405852c5a4249b8601d]
I0227 17:26:37.272168 16019 catalog_manager.cc:347] Loaded metadata for table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f]
I0227 17:26:37.272394 16019 catalog_manager.cc:347] Loaded metadata for table default.table_0 [id=bc75674adf8e42de8857bcba6d648867]
I0227 17:26:37.272568 16019 catalog_manager.cc:347] Loaded metadata for table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]
I0227 17:26:37.272738 16019 catalog_manager.cc:347] Loaded metadata for table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]
I0227 17:26:37.272912 16019 catalog_manager.cc:347] Loaded metadata for table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]
I0227 17:26:37.273095 16019 catalog_manager.cc:347] Loaded metadata for table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]
I0227 17:26:37.274406 16019 catalog_manager.cc:401] Loaded metadata for tablet 0156483f43f14f4e9fe0e5dc35c12333 (table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f])
I0227 17:26:37.274607 16019 catalog_manager.cc:401] Loaded metadata for tablet 363dd021fd184929bc8b1f461f8fdb16 (table default.table_2 [id=1386cdb42e004032916576ef12b3f5cd])
I0227 17:26:37.275161 16019 catalog_manager.cc:401] Loaded metadata for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276])
I0227 17:26:37.275362 16019 catalog_manager.cc:401] Loaded metadata for tablet 7fae960e7b5343338ea18dbb27e5f63a (table default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff])
I0227 17:26:37.275765 16019 catalog_manager.cc:401] Loaded metadata for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b])
I0227 17:26:37.275952 16019 catalog_manager.cc:401] Loaded metadata for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31])
I0227 17:26:37.276149 16019 catalog_manager.cc:401] Loaded metadata for tablet bf54e38e762844a18884463d41a59bf2 (table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45])
I0227 17:26:37.276341 16019 catalog_manager.cc:401] Loaded metadata for tablet dd3249f630b848c8b1efd3a68065de81 (table default.table_1 [id=7189818660584405852c5a4249b8601d])
I0227 17:26:37.276535 16019 catalog_manager.cc:401] Loaded metadata for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table default.table_0 [id=bc75674adf8e42de8857bcba6d648867])
I0227 17:26:37.276751 16019 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:26:37.278890 16019 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:26:37.279659 16019 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418: Loaded TSK: 0
I0227 17:26:37.279958 16019 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
W0227 17:26:37.803681 13447 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: connection negotiation to 127.112.245.254:35995 for RPC RequestConsensusVote timed out after 0.557s (ON_OUTBOUND_QUEUE)
I0227 17:26:37.803805 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:37.809303 16517 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I0227 17:26:37.858855 13605 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Refusing update from remote peer 6cd5de61888340e681a3e5cd19fad418: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 8 index: 44. (index mismatch)
I0227 17:26:37.859827 16515 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I0227 17:26:37.862017 16517 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd5de61888340e681a3e5cd19fad418. Latest consensus state: current_term: 8 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:37.862049 16517 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:37.871253 16515 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:37.871289 16515 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:37.871863 16517 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "6cd5de61888340e681a3e5cd19fad418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:37.871894 16517 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:26:37.894441 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.632s (SENT)
W0227 17:26:37.919847 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: RequestConsensusVote RPC to 127.112.245.254:35995 timed out after 0.660s (SENT)
I0227 17:26:38.074229 125069 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
I0227 17:26:38.129664 124856 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
I0227 17:26:38.130517 125383 heartbeater.cc:466] Master 127.112.245.253:50732 was elected leader, sending a full tablet report...
73432 T57 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
73433 T57 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
73433 T57 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
73434 T57 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
73434 T57 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
73435 T57 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
I0227 17:26:38.842224 17145 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:38.842346 17145 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:38.842864 17145 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:38.843312 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:38.843405 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:39.449615 13447 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: connection negotiation to 127.112.245.254:35995 for RPC RequestConsensusVote timed out after 0.605s (ON_OUTBOUND_QUEUE)
I0227 17:26:39.449731 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:39.450140 17805 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I0227 17:26:39.955652 18183 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:39.955771 18183 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:39.956426 18183 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:39.956670 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:39.956751 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:40.246781 16000 negotiation.cc:313] Failed RPC negotiation. Trace:
0227 17:26:37.245539 (+     0us) reactor.cc:583] Submitting negotiation task for client connection to 127.112.245.254:35995
0227 17:26:37.257720 (+ 12181us) negotiation.cc:98] Waiting for socket to connect
0227 17:26:37.257736 (+    16us) client_negotiation.cc:167] Beginning negotiation
0227 17:26:37.257828 (+    92us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request
0227 17:26:40.246189 (+2988361us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.112.245.254:35995
Metrics: {"client-negotiator.queue_time_us":12055,"mutex_wait_us":15,"thread_start_us":72,"threads_started":1}
W0227 17:26:40.246951 13447 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: Client connection negotiation failed: client connection to 127.112.245.254:35995
I0227 17:26:40.247022 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:40.247228 18183 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I0227 17:26:41.079809 20209 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:41.079929 20209 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:41.080505 20209 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:41.081146 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:41.081271 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:41.750957 13447 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: connection negotiation to 127.112.245.254:35995 for RPC RequestConsensusVote timed out after 0.668s (ON_OUTBOUND_QUEUE)
I0227 17:26:41.751045 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:41.751442 21012 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W0227 17:26:42.578749 125070 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:35995 (3 consecutive failures): Timed out: Failed to ping master at 127.112.245.254:35995: Ping RPC to 127.112.245.254:35995 timed out after 1.000s (SENT)
W0227 17:26:42.631433 125384 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:35995 (3 consecutive failures): Timed out: Failed to ping master at 127.112.245.254:35995: Ping RPC to 127.112.245.254:35995 timed out after 1.000s (SENT)
W0227 17:26:42.634672 124857 heartbeater.cc:586] Failed to heartbeat to 127.112.245.254:35995 (3 consecutive failures): Timed out: Failed to ping master at 127.112.245.254:35995: Ping RPC to 127.112.245.254:35995 timed out after 1.000s (SENT)
I0227 17:26:43.452811 22151 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:43.452973 22151 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:43.453687 22151 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:43.454213 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:43.454289 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:44.081290 20222 negotiation.cc:313] Failed RPC negotiation. Trace:
0227 17:26:41.080774 (+     0us) reactor.cc:583] Submitting negotiation task for client connection to 127.112.245.254:35995
0227 17:26:41.081122 (+   348us) negotiation.cc:98] Waiting for socket to connect
0227 17:26:41.081139 (+    17us) client_negotiation.cc:167] Beginning negotiation
0227 17:26:41.081253 (+   114us) client_negotiation.cc:244] Sending NEGOTIATE NegotiatePB request
0227 17:26:44.081160 (+2999907us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.112.245.254:35995
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
W0227 17:26:44.081421 13447 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: Client connection negotiation failed: client connection to 127.112.245.254:35995
I0227 17:26:44.081498 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:44.081841 22791 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I0227 17:26:44.937153 23942 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:44.937261 23942 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:44.937758 23942 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:44.938175 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:44.938249 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:45.479661 24149 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:45.479810 24149 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:45.480396 24149 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:45.480918 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:45.481019 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:45.565033 13447 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: connection negotiation to 127.112.245.254:35995 for RPC RequestConsensusVote timed out after 0.626s (ON_OUTBOUND_QUEUE)
I0227 17:26:45.565129 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:45.565335 24149 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W0227 17:26:46.116385 13447 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Timed out: connection negotiation to 127.112.245.254:35995 for RPC RequestConsensusVote timed out after 0.634s (ON_OUTBOUND_QUEUE)
I0227 17:26:46.116475 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9; no voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89
I0227 17:26:46.116788 24571 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W0227 17:26:46.821980 125414 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.254:35995)
I0227 17:26:46.826289 137450 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.253:35523:
name: "default.table_2_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:26:46.826684 137450 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:46.917953 124778 tablet_service.cc:774] Processing CreateTablet for tablet 4894d1b749c04d81a7faef34e4d7affc (table=default.table_2_0 [id=b8dd42b26d92481cba2426feb211b7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:46.918035 124988 tablet_service.cc:774] Processing CreateTablet for tablet 4894d1b749c04d81a7faef34e4d7affc (table=default.table_2_0 [id=b8dd42b26d92481cba2426feb211b7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:46.918170 125316 tablet_service.cc:774] Processing CreateTablet for tablet 4894d1b749c04d81a7faef34e4d7affc (table=default.table_2_0 [id=b8dd42b26d92481cba2426feb211b7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:46.918200 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 4894d1b749c04d81a7faef34e4d7affc. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:46.918257 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 4894d1b749c04d81a7faef34e4d7affc. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:46.918434 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 4894d1b749c04d81a7faef34e4d7affc. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:46.920182 124988 ts_tablet_manager.cc:1175] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:46.920187 124778 ts_tablet_manager.cc:1175] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:46.920426 25472 ts_tablet_manager.cc:1030] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:26:46.920426 25471 ts_tablet_manager.cc:1030] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:26:46.920548 25472 tablet_bootstrap.cc:439] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:26:46.920595 125316 ts_tablet_manager.cc:1175] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:46.920763 25471 tablet_bootstrap.cc:439] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:26:46.920821 25473 ts_tablet_manager.cc:1030] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:26:46.920908 25473 tablet_bootstrap.cc:439] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:26:46.921702 25472 tablet_bootstrap.cc:593] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:26:46.921754 25471 tablet_bootstrap.cc:593] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:26:46.921816 25473 tablet_bootstrap.cc:593] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:26:46.922789 25472 tablet_bootstrap.cc:439] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:26:46.922791 25471 tablet_bootstrap.cc:439] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:26:46.922812 25473 tablet_bootstrap.cc:439] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:26:46.922866 25472 ts_tablet_manager.cc:1047] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:26:46.922866 25471 ts_tablet_manager.cc:1047] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:26:46.922881 25473 ts_tablet_manager.cc:1047] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I0227 17:26:46.923362 25473 raft_consensus.cc:340] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.923374 25472 raft_consensus.cc:340] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.923374 25471 raft_consensus.cc:340] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.923393 25473 raft_consensus.cc:366] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:46.923400 25472 raft_consensus.cc:366] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:46.923401 25471 raft_consensus.cc:366] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:46.923415 25473 raft_consensus.cc:705] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:26:46.923421 25472 raft_consensus.cc:705] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:26:46.923422 25471 raft_consensus.cc:705] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:26:46.923549 25473 consensus_queue.cc:229] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.923553 25472 consensus_queue.cc:229] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.923553 25471 consensus_queue.cc:229] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.923804 25472 ts_tablet_manager.cc:1075] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:46.923977 25473 ts_tablet_manager.cc:1075] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:46.924036 25471 ts_tablet_manager.cc:1075] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:26:46.940847 25161 raft_consensus.cc:472] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:46.941002 25161 raft_consensus.cc:494] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.941601 25161 leader_election.cc:292] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:26:46.942193 124809 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4894d1b749c04d81a7faef34e4d7affc" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:26:46.942287 125339 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4894d1b749c04d81a7faef34e4d7affc" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:26:46.942313 124809 raft_consensus.cc:2346] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:26:46.942421 125339 raft_consensus.cc:2346] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:26:46.942771 124930 leader_election.cc:306] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:46.942924 25161 raft_consensus.cc:2640] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:26:46.942952 25161 raft_consensus.cc:472] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:26:46.942973 25161 raft_consensus.cc:2886] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:46.944195 25161 raft_consensus.cc:494] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.944600 25161 leader_election.cc:292] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Requested vote from peers 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806), 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942)
I0227 17:26:46.944957 124809 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4894d1b749c04d81a7faef34e4d7affc" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:26:46.945013 124809 raft_consensus.cc:2886] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:46.945051 125339 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "4894d1b749c04d81a7faef34e4d7affc" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:26:46.945175 125339 raft_consensus.cc:2886] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:46.946115 124809 raft_consensus.cc:2346] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:26:46.946468 124930 leader_election.cc:306] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:46.946595 25161 raft_consensus.cc:2640] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:26:46.946681 25161 raft_consensus.cc:667] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [term 1 LEADER]: Becoming Leader. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Running, Role: LEADER
I0227 17:26:46.946871 25161 consensus_queue.cc:206] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } }
I0227 17:26:46.946965 125339 raft_consensus.cc:2346] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:26:46.948673 137453 catalog_manager.cc:3791] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194). New cstate: current_term: 1 leader_uuid: "39144add463f4fe89dc1ea42c778d8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } }
W0227 17:26:46.960103 124966 tablet_service.cc:1915] Rejecting scan request for tablet 4894d1b749c04d81a7faef34e4d7affc: Uninitialized: safe time has not yet been initialized
W0227 17:26:47.161751 124966 tablet_service.cc:1915] Rejecting scan request for tablet 4894d1b749c04d81a7faef34e4d7affc: Uninitialized: safe time has not yet been initialized
I0227 17:26:47.344884 125339 raft_consensus.cc:1181] T 4894d1b749c04d81a7faef34e4d7affc P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:47.345474 25161 consensus_queue.cc:984] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:47.349653 124809 raft_consensus.cc:1181] T 4894d1b749c04d81a7faef34e4d7affc P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:47.350474 23041 consensus_queue.cc:984] T 4894d1b749c04d81a7faef34e4d7affc P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:47.453524 26286 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:47.453682 26286 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:47.454305 26286 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:47.454776 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 48 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:47.454854 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:47.565824 126662 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:47.565904 126662 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:47.565945 126661 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89"
I0227 17:26:47.566005 126661 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Leader election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 8 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:47.566112 126660 raft_consensus.cc:2881] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Stepping down as leader of term 7
I0227 17:26:47.566145 126660 raft_consensus.cc:705] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c31e774156b64c02922906e3e2985c89, State: Running, Role: LEADER
I0227 17:26:47.566300 126660 consensus_queue.cc:229] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:47.566334 126660 consensus_peers.cc:482] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 -> Peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Closing peer: 2e3d7e621b864672aaa91ea2d0bbd1b9
W0227 17:26:47.566356 26329 negotiation.cc:313] Failed RPC negotiation. Trace:
0227 17:26:47.565538 (+     0us) reactor.cc:583] Submitting negotiation task for server connection from 127.112.245.254:48699
0227 17:26:47.565709 (+   171us) server_negotiation.cc:183] Beginning negotiation
0227 17:26:47.565713 (+     4us) server_negotiation.cc:372] Waiting for connection header
0227 17:26:47.565730 (+    17us) server_negotiation.cc:380] Connection header received
0227 17:26:47.565774 (+    44us) server_negotiation.cc:336] Received NEGOTIATE NegotiatePB request
0227 17:26:47.565777 (+     3us) server_negotiation.cc:419] Received NEGOTIATE request from client
0227 17:26:47.565827 (+    50us) server_negotiation.cc:348] Sending NEGOTIATE NegotiatePB response
0227 17:26:47.565961 (+   134us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.112.245.254:48699: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":36,"threads_started":1}
I0227 17:26:47.566424 13605 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Rejecting Update request from peer c31e774156b64c02922906e3e2985c89 for earlier term 7. Current term is 8. Ops: []
I0227 17:26:47.566359 126660 consensus_peers.cc:482] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 -> Peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Closing peer: 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:47.566504 126660 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 7 FOLLOWER]: Advancing to term 8
W0227 17:26:47.566411 26327 negotiation.cc:313] Failed RPC negotiation. Trace:
0227 17:26:47.564991 (+     0us) reactor.cc:583] Submitting negotiation task for server connection from 127.112.245.254:48506
0227 17:26:47.565484 (+   493us) server_negotiation.cc:183] Beginning negotiation
0227 17:26:47.565490 (+     6us) server_negotiation.cc:372] Waiting for connection header
0227 17:26:47.565511 (+    21us) server_negotiation.cc:380] Connection header received
0227 17:26:47.565572 (+    61us) server_negotiation.cc:336] Received NEGOTIATE NegotiatePB request
0227 17:26:47.565577 (+     5us) server_negotiation.cc:419] Received NEGOTIATE request from client
0227 17:26:47.565649 (+    72us) server_negotiation.cc:348] Sending NEGOTIATE NegotiatePB response
0227 17:26:47.565838 (+   189us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.112.245.254:48506: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
I0227 17:26:47.567193 124857 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:26:47.567683 126660 raft_consensus.cc:1181] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 8 FOLLOWER]: Refusing update from remote peer 6cd5de61888340e681a3e5cd19fad418: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 8 index: 44. (index mismatch)
I0227 17:26:47.568320 126738 hms_notification_log_listener.cc:226] Skipping Hive Metastore notification log poll: Illegal state: Not the leader. Local UUID: c31e774156b64c02922906e3e2985c89, Raft Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:47.568373 125384 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
I0227 17:26:47.568454 126644 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.254:47010:
name: "default.table_2_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I0227 17:26:47.568572 125070 heartbeater.cc:345] Connected to a master server at 127.112.245.254:35995
W0227 17:26:47.568780 126644 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:47.570207 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 48 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:47.570288 126660 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 8.
I0227 17:26:47.570988 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89; no voters: 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:47.571173 26286 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I0227 17:26:47.571214 26286 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6cd5de61888340e681a3e5cd19fad418)
I0227 17:26:47.571244 26286 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 8 FOLLOWER]: Advancing to term 9
I0227 17:26:47.572463 26286 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:47.573004 26286 leader_election.cc:292] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 election: Requested vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732)
I0227 17:26:47.573209 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 48 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89"
I0227 17:26:47.573251 126660 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 8 FOLLOWER]: Advancing to term 9
I0227 17:26:47.574265 126660 raft_consensus.cc:2346] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 in term 9.
I0227 17:26:47.574503 13447 leader_election.cc:306] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:26:47.574645 26286 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 FOLLOWER]: Leader election won for term 9
I0227 17:26:47.574792 26286 raft_consensus.cc:667] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Becoming Leader. State: Replica: 2e3d7e621b864672aaa91ea2d0bbd1b9, State: Running, Role: LEADER
I0227 17:26:47.575158 26286 consensus_queue.cc:206] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 8.48, Last appended by leader: 48, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:47.576519 26350 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3d7e621b864672aaa91ea2d0bbd1b9. Latest consensus state: current_term: 9 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:47.576551 26350 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:47.576771 26355 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:26:47.578413 26355 catalog_manager.cc:347] Loaded metadata for table default.table_2 [id=1386cdb42e004032916576ef12b3f5cd]
I0227 17:26:47.578570 26355 catalog_manager.cc:347] Loaded metadata for table default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff]
I0227 17:26:47.578713 26355 catalog_manager.cc:347] Loaded metadata for table default.table_1 [id=7189818660584405852c5a4249b8601d]
I0227 17:26:47.578847 26355 catalog_manager.cc:347] Loaded metadata for table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f]
I0227 17:26:47.578984 26355 catalog_manager.cc:347] Loaded metadata for table default.table_2_0 [id=b8dd42b26d92481cba2426feb211b7c5]
I0227 17:26:47.579139 26355 catalog_manager.cc:347] Loaded metadata for table default.table_0 [id=bc75674adf8e42de8857bcba6d648867]
I0227 17:26:47.579286 26355 catalog_manager.cc:347] Loaded metadata for table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]
I0227 17:26:47.579421 26355 catalog_manager.cc:347] Loaded metadata for table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]
I0227 17:26:47.579563 26355 catalog_manager.cc:347] Loaded metadata for table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]
I0227 17:26:47.579699 26355 catalog_manager.cc:347] Loaded metadata for table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]
I0227 17:26:47.580917 26355 catalog_manager.cc:401] Loaded metadata for tablet 0156483f43f14f4e9fe0e5dc35c12333 (table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f])
I0227 17:26:47.581084 26355 catalog_manager.cc:401] Loaded metadata for tablet 363dd021fd184929bc8b1f461f8fdb16 (table default.table_2 [id=1386cdb42e004032916576ef12b3f5cd])
I0227 17:26:47.581269 26355 catalog_manager.cc:401] Loaded metadata for tablet 4894d1b749c04d81a7faef34e4d7affc (table default.table_2_0 [id=b8dd42b26d92481cba2426feb211b7c5])
I0227 17:26:47.581450 26355 catalog_manager.cc:401] Loaded metadata for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276])
I0227 17:26:47.581607 26355 catalog_manager.cc:401] Loaded metadata for tablet 7fae960e7b5343338ea18dbb27e5f63a (table default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff])
I0227 17:26:47.581753 26355 catalog_manager.cc:401] Loaded metadata for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b])
I0227 17:26:47.581898 26355 catalog_manager.cc:401] Loaded metadata for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31])
I0227 17:26:47.582044 26355 catalog_manager.cc:401] Loaded metadata for tablet bf54e38e762844a18884463d41a59bf2 (table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45])
I0227 17:26:47.582204 26355 catalog_manager.cc:401] Loaded metadata for tablet dd3249f630b848c8b1efd3a68065de81 (table default.table_1 [id=7189818660584405852c5a4249b8601d])
I0227 17:26:47.582348 26355 catalog_manager.cc:401] Loaded metadata for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table default.table_0 [id=bc75674adf8e42de8857bcba6d648867])
I0227 17:26:47.582463 26355 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:26:47.583468 26355 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:26:47.584192 26355 catalog_manager.cc:4106] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9: Loaded TSK: 0
I0227 17:26:47.584743 26355 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
17:26:47.588 [HMS - ERROR - pool-8-thread-2] (RetryingHMSHandler.java:204) AlreadyExistsException(message:Table table_2_0 already exists)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1452)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1539)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
	at com.sun.proxy.$Proxy24.create_table_with_environment_context(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11191)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11175)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

W0227 17:26:47.589220 126644 catalog_manager.cc:1615] Already present: an error occurred while creating table default.table_2_0 in the HMS: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table table_2_0 already exists)
I0227 17:26:47.908607 125068 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
W0227 17:26:48.080070 13448 leader_election.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.506s (SENT)
I0227 17:26:48.093420 26849 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:48.093622 26849 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:48.094255 26849 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:48.094727 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:48.094811 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:48.108574 126660 raft_consensus.cc:1181] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Refusing update from remote peer 2e3d7e621b864672aaa91ea2d0bbd1b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 9 index: 49. (index mismatch)
I0227 17:26:48.109031 26350 consensus_queue.cc:984] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 43, Time since last communication: 0.000s
I0227 17:26:48.112223 26849 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3d7e621b864672aaa91ea2d0bbd1b9. Latest consensus state: current_term: 9 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:48.112264 26849 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:48.112498 26857 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:48.112524 26857 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:48.113194 26857 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:48.113203 26350 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:48.113221 26857 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:48.113229 26350 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:48.352078 125382 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
I0227 17:26:48.355402 124855 heartbeater.cc:466] Master 127.112.245.252:58453 was elected leader, sending a full tablet report...
W0227 17:26:48.630424 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.535s (SENT)
I0227 17:26:48.630515 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:48.630822 27613 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I0227 17:26:49.973273 30202 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:49.973443 30202 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:49.974035 30202 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:49.974602 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:49.974675 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:50.491536 31687 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:50.491757 31687 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:50.492496 31687 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:50.493068 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:50.493154 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:50.504098 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.529s (SENT)
I0227 17:26:50.504186 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:50.504410 31687 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W0227 17:26:50.997951 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.504s (SENT)
I0227 17:26:50.998021 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:50.998214 31687 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I0227 17:26:52.100412 35022 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:52.100554 35022 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:52.101202 35022 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:52.101725 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:52.101864 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:52.719897 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.617s (SENT)
I0227 17:26:52.720026 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:52.720412 36193 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I0227 17:26:53.826573 38026 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:53.826705 38026 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:53.827493 38026 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:53.827888 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:53.827973 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:54.434980 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.607s (SENT)
I0227 17:26:54.435077 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:54.444947 38732 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I0227 17:26:55.266199 39350 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:55.266346 39350 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:55.266876 39350 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:55.267426 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:55.267513 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
W0227 17:26:55.912118 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.644s (SENT)
I0227 17:26:55.912204 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:55.916347 39713 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W0227 17:26:57.577251 125415 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.253:50732)
I0227 17:26:57.583679 13602 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.252:51672:
name: "default.table_2_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:26:57.585536 13602 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:26:57.637347 41069 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:57.637537 41069 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:57.638697 41069 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:57.639322 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:26:57.639415 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:57.704727 124988 tablet_service.cc:774] Processing CreateTablet for tablet ecb4dca085d84488941dd64175a6aaac (table=default.table_2_1 [id=d603b950501d4d128e92498f10ad7ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:57.704977 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet ecb4dca085d84488941dd64175a6aaac. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:57.705467 125316 tablet_service.cc:774] Processing CreateTablet for tablet ecb4dca085d84488941dd64175a6aaac (table=default.table_2_1 [id=d603b950501d4d128e92498f10ad7ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:57.705699 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet ecb4dca085d84488941dd64175a6aaac. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:57.708648 124988 ts_tablet_manager.cc:1175] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:57.708905 41149 ts_tablet_manager.cc:1030] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:26:57.708993 41149 tablet_bootstrap.cc:439] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:26:57.709836 124778 tablet_service.cc:774] Processing CreateTablet for tablet ecb4dca085d84488941dd64175a6aaac (table=default.table_2_1 [id=d603b950501d4d128e92498f10ad7ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:26:57.710068 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet ecb4dca085d84488941dd64175a6aaac. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:26:57.710098 41149 tablet_bootstrap.cc:593] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:26:57.710237 125316 ts_tablet_manager.cc:1175] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:57.710497 41152 ts_tablet_manager.cc:1030] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:26:57.710602 41152 tablet_bootstrap.cc:439] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:26:57.711474 41152 tablet_bootstrap.cc:593] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:26:57.712713 41149 tablet_bootstrap.cc:439] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:26:57.712811 41149 ts_tablet_manager.cc:1047] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I0227 17:26:57.713430 41149 raft_consensus.cc:340] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.713457 41149 raft_consensus.cc:366] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:57.713479 41149 raft_consensus.cc:705] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:26:57.713572 124778 ts_tablet_manager.cc:1175] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:26:57.713618 41149 consensus_queue.cc:229] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.713743 41152 tablet_bootstrap.cc:439] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:26:57.713809 41152 ts_tablet_manager.cc:1047] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I0227 17:26:57.713891 41149 ts_tablet_manager.cc:1075] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0227 17:26:57.714119 41155 ts_tablet_manager.cc:1030] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:26:57.714226 41155 tablet_bootstrap.cc:439] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:26:57.714257 41152 raft_consensus.cc:340] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.714280 41152 raft_consensus.cc:366] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:57.714300 41152 raft_consensus.cc:705] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:26:57.714403 41152 consensus_queue.cc:229] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.714610 41152 ts_tablet_manager.cc:1075] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I0227 17:26:57.715209 41155 tablet_bootstrap.cc:593] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:26:57.717250 41155 tablet_bootstrap.cc:439] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:26:57.717317 41155 ts_tablet_manager.cc:1047] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I0227 17:26:57.717891 41155 raft_consensus.cc:340] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.717922 41155 raft_consensus.cc:366] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:26:57.717944 41155 raft_consensus.cc:705] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:26:57.718098 41155 consensus_queue.cc:229] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.718358 41155 ts_tablet_manager.cc:1075] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
W0227 17:26:57.731248 125071 tablet.cc:1786] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:26:57.738127 124858 tablet.cc:1786] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338: Can't schedule compaction. Clean time has not been advanced past its initial value.
W0227 17:26:57.749017 125385 tablet.cc:1786] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180: Can't schedule compaction. Clean time has not been advanced past its initial value.
I0227 17:26:57.780552 38888 raft_consensus.cc:472] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:26:57.780676 38888 raft_consensus.cc:494] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.781198 38888 leader_election.cc:292] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:26:57.781790 124809 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "ecb4dca085d84488941dd64175a6aaac" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:26:57.781818 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "ecb4dca085d84488941dd64175a6aaac" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:26:57.781886 124809 raft_consensus.cc:2346] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:26:57.781903 125337 raft_consensus.cc:2346] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:26:57.782331 124930 leader_election.cc:306] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:57.782591 38888 raft_consensus.cc:2640] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:26:57.782619 38888 raft_consensus.cc:472] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:26:57.782639 38888 raft_consensus.cc:2886] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:57.784495 38888 raft_consensus.cc:494] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.784905 38888 leader_election.cc:292] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Requested vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:26:57.785372 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "ecb4dca085d84488941dd64175a6aaac" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:26:57.785436 125337 raft_consensus.cc:2886] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:57.785632 124809 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "ecb4dca085d84488941dd64175a6aaac" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:26:57.785709 124809 raft_consensus.cc:2886] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:26:57.787117 124809 raft_consensus.cc:2346] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:26:57.787569 125337 raft_consensus.cc:2346] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:26:57.787605 124930 leader_election.cc:306] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:26:57.787739 38888 raft_consensus.cc:2640] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:26:57.787958 38888 raft_consensus.cc:667] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [term 1 LEADER]: Becoming Leader. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Running, Role: LEADER
I0227 17:26:57.788106 38888 consensus_queue.cc:206] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } }
I0227 17:26:57.790928 13601 catalog_manager.cc:3791] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194). New cstate: current_term: 1 leader_uuid: "39144add463f4fe89dc1ea42c778d8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } }
W0227 17:26:57.822461 124966 tablet_service.cc:1915] Rejecting scan request for tablet ecb4dca085d84488941dd64175a6aaac: Uninitialized: safe time has not yet been initialized
W0227 17:26:58.024226 124966 tablet_service.cc:1915] Rejecting scan request for tablet ecb4dca085d84488941dd64175a6aaac: Uninitialized: safe time has not yet been initialized
I0227 17:26:58.241076 124809 raft_consensus.cc:1181] T ecb4dca085d84488941dd64175a6aaac P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:58.241678 38888 consensus_queue.cc:984] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:26:58.249159 125339 raft_consensus.cc:1181] T ecb4dca085d84488941dd64175a6aaac P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:26:58.249729 41809 consensus_queue.cc:984] T ecb4dca085d84488941dd64175a6aaac P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W0227 17:26:58.280733 126593 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Timed out: RequestConsensusVote RPC to 127.112.245.253:50732 timed out after 0.641s (SENT)
I0227 17:26:58.280838 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31e774156b64c02922906e3e2985c89; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, 6cd5de61888340e681a3e5cd19fad418
I0227 17:26:58.281536 41837 raft_consensus.cc:2588] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I0227 17:26:58.431728 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" candidate_term: 9 candidate_status { last_received { term: 8 index: 48 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418"
I0227 17:26:58.431814 137461 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Leader election vote request: Denying vote to candidate 2e3d7e621b864672aaa91ea2d0bbd1b9 for term 9 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:58.431990 137457 raft_consensus.cc:1149] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Rejecting Update request from peer c31e774156b64c02922906e3e2985c89 for earlier term 7. Current term is 8. Ops: []
I0227 17:26:58.432093 137462 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.432119 137456 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.432137 137461 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.432142 137455 raft_consensus.cc:2881] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Stepping down as leader of term 8
I0227 17:26:58.432186 137455 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6cd5de61888340e681a3e5cd19fad418, State: Running, Role: LEADER
I0227 17:26:58.432312 137457 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.432360 137455 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 8.48, Last appended by leader: 48, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:58.432443 137463 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.432540 137458 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.432718 137459 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 50 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.432862 42168 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I0227 17:26:58.432868 137455 consensus_peers.cc:482] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 -> Peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Closing peer: 2e3d7e621b864672aaa91ea2d0bbd1b9
I0227 17:26:58.432937 137455 consensus_peers.cc:482] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 -> Peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Closing peer: c31e774156b64c02922906e3e2985c89
I0227 17:26:58.433048 137455 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 8 FOLLOWER]: Advancing to term 9
I0227 17:26:58.434192 137455 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Refusing update from remote peer 2e3d7e621b864672aaa91ea2d0bbd1b9: Log matching property violated. Preceding OpId in replica: term: 8 index: 48. Preceding OpId from leader: term: 9 index: 49. (index mismatch)
I0227 17:26:58.434270 137462 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:58.434669 137461 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:58.434733 137463 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:58.434794 137458 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:58.434849 137459 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
W0227 17:26:58.434926 137503 hms_notification_log_listener.cc:130] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events
I0227 17:26:58.435056 137457 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:58.435099 124856 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:58.435257 137456 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
I0227 17:26:58.444784 125069 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:58.445508 125383 heartbeater.cc:345] Connected to a master server at 127.112.245.253:50732
I0227 17:26:58.784453 42706 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:58.784574 42706 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:58.785050 42706 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:58.785471 137456 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 53 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418" is_pre_election: true
I0227 17:26:58.785565 137456 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 9.
I0227 17:26:58.785861 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:26:58.786025 42706 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Leader pre-election won for term 10
I0227 17:26:58.786108 42706 raft_consensus.cc:472] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2e3d7e621b864672aaa91ea2d0bbd1b9)
I0227 17:26:58.786175 42706 raft_consensus.cc:2886] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 9 FOLLOWER]: Advancing to term 10
I0227 17:26:58.787792 42706 raft_consensus.cc:494] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:58.788300 42706 leader_election.cc:292] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 election: Requested vote from peers 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:58.788666 137456 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 53 } } ignore_live_leader: false dest_uuid: "6cd5de61888340e681a3e5cd19fad418"
I0227 17:26:58.788777 137456 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 9 FOLLOWER]: Advancing to term 10
I0227 17:26:58.790310 137456 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31e774156b64c02922906e3e2985c89 in term 10.
I0227 17:26:58.790588 126593 leader_election.cc:306] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418, c31e774156b64c02922906e3e2985c89; no voters: 
I0227 17:26:58.790760 42706 raft_consensus.cc:2640] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 FOLLOWER]: Leader election won for term 10
I0227 17:26:58.790967 42706 raft_consensus.cc:667] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Becoming Leader. State: Replica: c31e774156b64c02922906e3e2985c89, State: Running, Role: LEADER
I0227 17:26:58.791275 42706 consensus_queue.cc:206] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 9.53, Last appended by leader: 53, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:58.794368 42750 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 10 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:58.794777 42750 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
I0227 17:26:58.795001 42776 catalog_manager.cc:1046] Loading table and tablet metadata into memory...
I0227 17:26:58.796814 42776 catalog_manager.cc:347] Loaded metadata for table default.table_2 [id=1386cdb42e004032916576ef12b3f5cd]
I0227 17:26:58.797013 42776 catalog_manager.cc:347] Loaded metadata for table default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff]
I0227 17:26:58.797171 42776 catalog_manager.cc:347] Loaded metadata for table default.table_1 [id=7189818660584405852c5a4249b8601d]
I0227 17:26:58.797319 42776 catalog_manager.cc:347] Loaded metadata for table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f]
I0227 17:26:58.797495 42776 catalog_manager.cc:347] Loaded metadata for table default.table_2_0 [id=b8dd42b26d92481cba2426feb211b7c5]
I0227 17:26:58.797667 42776 catalog_manager.cc:347] Loaded metadata for table default.table_0 [id=bc75674adf8e42de8857bcba6d648867]
I0227 17:26:58.797842 42776 catalog_manager.cc:347] Loaded metadata for table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45]
I0227 17:26:58.798017 42776 catalog_manager.cc:347] Loaded metadata for table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31]
I0227 17:26:58.798204 42776 catalog_manager.cc:347] Loaded metadata for table default.table_2_1 [id=d603b950501d4d128e92498f10ad7ee7]
I0227 17:26:58.798377 42776 catalog_manager.cc:347] Loaded metadata for table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b]
I0227 17:26:58.798566 42776 catalog_manager.cc:347] Loaded metadata for table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276]
I0227 17:26:58.799942 42776 catalog_manager.cc:401] Loaded metadata for tablet 0156483f43f14f4e9fe0e5dc35c12333 (table default.table_1_1 [id=ac73b8b72e334b67b7a5ce715797317f])
I0227 17:26:58.800140 42776 catalog_manager.cc:401] Loaded metadata for tablet 363dd021fd184929bc8b1f461f8fdb16 (table default.table_2 [id=1386cdb42e004032916576ef12b3f5cd])
I0227 17:26:58.800333 42776 catalog_manager.cc:401] Loaded metadata for tablet 4894d1b749c04d81a7faef34e4d7affc (table default.table_2_0 [id=b8dd42b26d92481cba2426feb211b7c5])
I0227 17:26:58.800520 42776 catalog_manager.cc:401] Loaded metadata for tablet 5c60704670c44ebf81ffea4c83d5a8d4 (table default.table_1_0 [id=fc09c0e69c204983ad92186871b2f276])
I0227 17:26:58.800705 42776 catalog_manager.cc:401] Loaded metadata for tablet 7fae960e7b5343338ea18dbb27e5f63a (table default.table_1_2 [id=6e03f2b32ac4487aaefa599456dee7ff])
I0227 17:26:58.800892 42776 catalog_manager.cc:401] Loaded metadata for tablet 8c4950a6e6e54bc59d6e40c00df12fab (table default.table_0_1 [id=d8ec01a9c6424f27a548a16c9f453c4b])
I0227 17:26:58.801091 42776 catalog_manager.cc:401] Loaded metadata for tablet 9808ecd2a4be428290fa53bb51dd17d4 (table default.table_0_0 [id=d18d4af2ae47433f9af25b21a92b3c31])
I0227 17:26:58.801275 42776 catalog_manager.cc:401] Loaded metadata for tablet bf54e38e762844a18884463d41a59bf2 (table default.table_0_2 [id=c00fb13ccb9d43a6872d134e76f8bb45])
I0227 17:26:58.801460 42776 catalog_manager.cc:401] Loaded metadata for tablet dd3249f630b848c8b1efd3a68065de81 (table default.table_1 [id=7189818660584405852c5a4249b8601d])
I0227 17:26:58.801642 42776 catalog_manager.cc:401] Loaded metadata for tablet ecb4dca085d84488941dd64175a6aaac (table default.table_2_1 [id=d603b950501d4d128e92498f10ad7ee7])
I0227 17:26:58.801831 42776 catalog_manager.cc:401] Loaded metadata for tablet fcd66b3d06b5442b9b1b4d34c6d0667b (table default.table_0 [id=bc75674adf8e42de8857bcba6d648867])
I0227 17:26:58.802045 42776 catalog_manager.cc:1057] Initializing Kudu internal certificate authority...
I0227 17:26:58.803221 42776 catalog_manager.cc:1066] Loading token signing keys...
I0227 17:26:58.804028 42776 catalog_manager.cc:4106] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89: Loaded TSK: 0
I0227 17:26:58.804306 42776 catalog_manager.cc:1077] Loading latest processed Hive Metastore notification log event ID...
I0227 17:26:59.293148 137456 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Refusing update from remote peer c31e774156b64c02922906e3e2985c89: Log matching property violated. Preceding OpId in replica: term: 8 index: 48. Preceding OpId from leader: term: 10 index: 54. (index mismatch)
I0227 17:26:59.293839 42750 consensus_queue.cc:984] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 48, Time since last communication: 0.000s
I0227 17:26:59.296510 43370 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:59.296576 43367 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 10 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:59.296644 43370 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:59.296666 43367 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:59.298912 43367 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:59.298933 43367 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:26:59.299321 42750 sys_catalog.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:26:59.299357 42750 sys_catalog.cc:340] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [sys.catalog]: This master's current role is: LEADER
W0227 17:26:59.412225 126592 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.625s (SENT)
W0227 17:26:59.445611 126592 leader_election.cc:337] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.656s (SENT)
I0227 17:26:59.754422 124857 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:26:59.757334 125384 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:26:59.799921 43919 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:26:59.800102 43919 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:26:59.800823 43919 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:26:59.801271 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:26:59.801345 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:26:59.805431 125070 heartbeater.cc:466] Master 127.112.245.254:35995 was elected leader, sending a full tablet report...
I0227 17:27:00.311964 45030 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:27:00.312096 45030 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:00.313813 45030 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:27:00.314281 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:27:00.314366 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
W0227 17:27:00.334749 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.529s (SENT)
I0227 17:27:00.334826 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:00.334998 45030 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I0227 17:27:00.844394 46579 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:27:00.844615 46579 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:00.845468 46579 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:27:00.845887 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:27:00.845975 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
W0227 17:27:01.042392 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.727s (SENT)
I0227 17:27:01.042476 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:01.042673 46579 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W0227 17:27:01.586840 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.740s (SENT)
I0227 17:27:01.586913 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:01.587353 47870 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I0227 17:27:02.392701 48314 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:27:02.392949 48314 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:02.393889 48314 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:27:02.393919 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:27:02.393982 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
97671 T50 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
97671 T50 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
97672 T50 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
97672 T50 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
97672 T50 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
97673 T50 odu.Log4JLogger.info Request to load fields "comment,name,type" of class org.apache.hadoop.hive.metastore.model.MFieldSchema but object is embedded, so ignored
W0227 17:27:03.142356 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.747s (SENT)
I0227 17:27:03.142455 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:03.142825 48731 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I0227 17:27:04.143780 49842 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:27:04.144059 49842 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:04.145237 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:27:04.145315 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:04.145422 49842 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
W0227 17:27:04.716338 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.570s (SENT)
I0227 17:27:04.716439 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:04.724382 50286 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I0227 17:27:05.254729 51088 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:27:05.254875 51088 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:05.255406 51088 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:27:05.259980 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:27:05.260071 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
W0227 17:27:06.285158 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 1.024s (SENT)
I0227 17:27:06.285250 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:06.290246 52040 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I0227 17:27:07.164427 52868 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:27:07.164600 52868 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:07.165513 52868 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:27:07.166435 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:27:07.166508 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:07.673934 53524 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c31e774156b64c02922906e3e2985c89)
I0227 17:27:07.674142 53524 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:07.676640 53524 leader_election.cc:292] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995), 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453)
I0227 17:27:07.677350 126660 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "c31e774156b64c02922906e3e2985c89" is_pre_election: true
I0227 17:27:07.677487 126660 raft_consensus.cc:2307] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
W0227 17:27:07.754575 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.588s (SENT)
I0227 17:27:07.754683 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:07.754896 53524 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W0227 17:27:08.415900 137432 leader_election.cc:337] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2e3d7e621b864672aaa91ea2d0bbd1b9 (127.112.245.252:58453): Timed out: RequestConsensusVote RPC to 127.112.245.252:58453 timed out after 0.738s (SENT)
I0227 17:27:08.416013 137432 leader_election.cc:306] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cd5de61888340e681a3e5cd19fad418; no voters: 2e3d7e621b864672aaa91ea2d0bbd1b9, c31e774156b64c02922906e3e2985c89
I0227 17:27:08.416393 54468 raft_consensus.cc:2588] T 00000000000000000000000000000000 P 6cd5de61888340e681a3e5cd19fad418 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W0227 17:27:08.444545 125414 master_proxy_rpc.cc:192] Re-attempting CreateTable request to leader Master (127.112.245.252:58453)
I0227 17:27:08.448936 126644 catalog_manager.cc:1386] Servicing CreateTable request from {username='adar'} at 127.112.245.254:47010:
name: "default.table_2_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W0227 17:27:08.449339 126644 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I0227 17:27:08.534425 125316 tablet_service.cc:774] Processing CreateTablet for tablet 0ffbcc0cc3e246d58b8c632013df602e (table=default.table_2_2 [id=75b302f74978469e9d008e2d91fd3c45]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:27:08.534593 124778 tablet_service.cc:774] Processing CreateTablet for tablet 0ffbcc0cc3e246d58b8c632013df602e (table=default.table_2_2 [id=75b302f74978469e9d008e2d91fd3c45]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:27:08.534711 125316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0ffbcc0cc3e246d58b8c632013df602e. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:27:08.534831 124778 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0ffbcc0cc3e246d58b8c632013df602e. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:27:08.534943 124988 tablet_service.cc:774] Processing CreateTablet for tablet 0ffbcc0cc3e246d58b8c632013df602e (table=default.table_2_2 [id=75b302f74978469e9d008e2d91fd3c45]), partition=RANGE (key) PARTITION UNBOUNDED
I0227 17:27:08.535210 124988 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 0ffbcc0cc3e246d58b8c632013df602e. 1 dirs total, 0 dirs full, 0 dirs failed
I0227 17:27:08.547894 124988 ts_tablet_manager.cc:1175] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:27:08.547901 125316 ts_tablet_manager.cc:1175] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:27:08.547902 124778 ts_tablet_manager.cc:1175] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338: Registered tablet (data state: TABLET_DATA_READY)
I0227 17:27:08.548156 54514 ts_tablet_manager.cc:1030] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180: Bootstrapping tablet
I0227 17:27:08.548184 54515 ts_tablet_manager.cc:1030] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf: Bootstrapping tablet
I0227 17:27:08.548259 54514 tablet_bootstrap.cc:439] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180: Bootstrap starting.
I0227 17:27:08.548276 54515 tablet_bootstrap.cc:439] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf: Bootstrap starting.
I0227 17:27:08.548413 54513 ts_tablet_manager.cc:1030] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338: Bootstrapping tablet
I0227 17:27:08.548518 54513 tablet_bootstrap.cc:439] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338: Bootstrap starting.
I0227 17:27:08.549659 54513 tablet_bootstrap.cc:593] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338: No blocks or log segments found. Creating new log.
I0227 17:27:08.549665 54515 tablet_bootstrap.cc:593] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf: No blocks or log segments found. Creating new log.
I0227 17:27:08.549696 54514 tablet_bootstrap.cc:593] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180: No blocks or log segments found. Creating new log.
I0227 17:27:08.550731 54514 tablet_bootstrap.cc:439] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180: No bootstrap required, opened a new log
I0227 17:27:08.550807 54514 ts_tablet_manager.cc:1047] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I0227 17:27:08.550842 54515 tablet_bootstrap.cc:439] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf: No bootstrap required, opened a new log
I0227 17:27:08.550912 54515 ts_tablet_manager.cc:1047] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I0227 17:27:08.551232 54514 raft_consensus.cc:340] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.551254 54514 raft_consensus.cc:366] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:27:08.551265 54514 raft_consensus.cc:705] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f55686db84876a066f455f6a41180, State: Initialized, Role: FOLLOWER
I0227 17:27:08.551283 54513 tablet_bootstrap.cc:439] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338: No bootstrap required, opened a new log
I0227 17:27:08.551388 54514 consensus_queue.cc:229] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.551414 54513 ts_tablet_manager.cc:1047] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I0227 17:27:08.551463 54515 raft_consensus.cc:340] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.551492 54515 raft_consensus.cc:366] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:27:08.551513 54515 raft_consensus.cc:705] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Initialized, Role: FOLLOWER
I0227 17:27:08.551631 54514 ts_tablet_manager.cc:1075] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:27:08.551648 54515 consensus_queue.cc:229] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.551897 54515 ts_tablet_manager.cc:1075] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I0227 17:27:08.552266 54513 raft_consensus.cc:340] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.552326 54513 raft_consensus.cc:366] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I0227 17:27:08.552367 54513 raft_consensus.cc:705] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6459ebb80643b98e01844aa2fd6338, State: Initialized, Role: FOLLOWER
I0227 17:27:08.552630 54513 consensus_queue.cc:229] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.553069 54513 ts_tablet_manager.cc:1075] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I0227 17:27:08.799774 50450 raft_consensus.cc:472] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I0227 17:27:08.799908 50450 raft_consensus.cc:494] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.800462 50450 leader_election.cc:292] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:27:08.800987 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0ffbcc0cc3e246d58b8c632013df602e" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338" is_pre_election: true
I0227 17:27:08.801087 124804 raft_consensus.cc:2346] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:27:08.801173 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0ffbcc0cc3e246d58b8c632013df602e" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180" is_pre_election: true
I0227 17:27:08.801256 125337 raft_consensus.cc:2346] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 0.
I0227 17:27:08.801551 124930 leader_election.cc:306] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:27:08.802018 50450 raft_consensus.cc:2640] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I0227 17:27:08.802043 50450 raft_consensus.cc:472] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I0227 17:27:08.802062 50450 raft_consensus.cc:2886] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 0 FOLLOWER]: Advancing to term 1
I0227 17:27:08.803226 50450 raft_consensus.cc:494] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.803583 50450 leader_election.cc:292] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Requested vote from peers 9a4f55686db84876a066f455f6a41180 (127.112.245.195:45942), 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806)
I0227 17:27:08.804147 125337 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0ffbcc0cc3e246d58b8c632013df602e" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4f55686db84876a066f455f6a41180"
I0227 17:27:08.804198 125337 raft_consensus.cc:2886] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:27:08.804234 124804 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "0ffbcc0cc3e246d58b8c632013df602e" candidate_uuid: "39144add463f4fe89dc1ea42c778d8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6459ebb80643b98e01844aa2fd6338"
I0227 17:27:08.804280 124804 raft_consensus.cc:2886] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [term 0 FOLLOWER]: Advancing to term 1
I0227 17:27:08.805291 124804 raft_consensus.cc:2346] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:27:08.805303 125337 raft_consensus.cc:2346] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39144add463f4fe89dc1ea42c778d8cf in term 1.
I0227 17:27:08.805902 124930 leader_election.cc:306] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39144add463f4fe89dc1ea42c778d8cf, 9f6459ebb80643b98e01844aa2fd6338; no voters: 
I0227 17:27:08.806274 50450 raft_consensus.cc:2640] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 1 FOLLOWER]: Leader election won for term 1
I0227 17:27:08.806473 50450 raft_consensus.cc:667] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [term 1 LEADER]: Becoming Leader. State: Replica: 39144add463f4fe89dc1ea42c778d8cf, State: Running, Role: LEADER
I0227 17:27:08.806718 50450 consensus_queue.cc:206] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } }
I0227 17:27:08.809011 126652 catalog_manager.cc:3791] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 39144add463f4fe89dc1ea42c778d8cf (127.112.245.194). New cstate: current_term: 1 leader_uuid: "39144add463f4fe89dc1ea42c778d8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39144add463f4fe89dc1ea42c778d8cf" member_type: VOTER last_known_addr { host: "127.112.245.194" port: 55958 } health_report { overall_health: HEALTHY } } }
W0227 17:27:08.828734 124966 tablet_service.cc:1915] Rejecting scan request for tablet 0ffbcc0cc3e246d58b8c632013df602e: Uninitialized: safe time has not yet been initialized
W0227 17:27:09.030496 124966 tablet_service.cc:1915] Rejecting scan request for tablet 0ffbcc0cc3e246d58b8c632013df602e: Uninitialized: safe time has not yet been initialized
I0227 17:27:09.278029 124804 raft_consensus.cc:1181] T 0ffbcc0cc3e246d58b8c632013df602e P 9f6459ebb80643b98e01844aa2fd6338 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:27:09.278965 50450 consensus_queue.cc:984] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6459ebb80643b98e01844aa2fd6338" member_type: VOTER last_known_addr { host: "127.112.245.193" port: 55806 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:27:09.285691 125337 raft_consensus.cc:1181] T 0ffbcc0cc3e246d58b8c632013df602e P 9a4f55686db84876a066f455f6a41180 [term 1 FOLLOWER]: Refusing update from remote peer 39144add463f4fe89dc1ea42c778d8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I0227 17:27:09.286394 55560 consensus_queue.cc:984] T 0ffbcc0cc3e246d58b8c632013df602e P 39144add463f4fe89dc1ea42c778d8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4f55686db84876a066f455f6a41180" member_type: VOTER last_known_addr { host: "127.112.245.195" port: 45942 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I0227 17:27:09.435935 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 124249
I0227 17:27:09.436019 13605 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.436120 13605 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.436189 13612 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.436255 13612 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.436352 13609 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.436396 13609 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.436664 13609 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.436702 13609 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.436831 13608 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 53 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.436877 13608 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.436944 13609 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.436985 13609 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.437113 13612 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 54 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.437156 13612 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.437156 13609 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.437218 13609 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.437355 13612 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cd5de61888340e681a3e5cd19fad418" candidate_term: 11 candidate_status { last_received { term: 10 index: 55 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" is_pre_election: true
I0227 17:27:09.437402 13612 raft_consensus.cc:2307] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Leader pre-election vote request: Denying vote to candidate 6cd5de61888340e681a3e5cd19fad418 for term 11 because replica is either leader or believes a valid leader to be alive.
I0227 17:27:09.437495 13606 tablet_service.cc:1010] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31e774156b64c02922906e3e2985c89" candidate_term: 10 candidate_status { last_received { term: 9 index: 53 } } ignore_live_leader: false dest_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9"
I0227 17:27:09.437513 13605 raft_consensus.cc:2881] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Stepping down as leader of term 9
I0227 17:27:09.437535 13605 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2e3d7e621b864672aaa91ea2d0bbd1b9, State: Running, Role: LEADER
I0227 17:27:09.437722 13605 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 9.53, Last appended by leader: 53, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } }
I0227 17:27:09.437762 13605 consensus_peers.cc:482] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 -> Peer 6cd5de61888340e681a3e5cd19fad418 (127.112.245.253:50732): Closing peer: 6cd5de61888340e681a3e5cd19fad418
I0227 17:27:09.437873 13605 consensus_peers.cc:482] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 -> Peer c31e774156b64c02922906e3e2985c89 (127.112.245.254:35995): Closing peer: c31e774156b64c02922906e3e2985c89
W0227 17:27:09.437974 55954 consensus_queue.cc:1099] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6cd5de61888340e681a3e5cd19fad418" responder_term: 9 status { last_received { term: 8 index: 48 } last_committed_idx: 48 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 8 index: 48. Preceding OpId from leader: term: 9 index: 49. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I0227 17:27:09.437992 13605 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 9 FOLLOWER]: Advancing to term 10
I0227 17:27:09.438740 125068 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:27:09.439038 13605 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 10 FOLLOWER]: Refusing update from remote peer c31e774156b64c02922906e3e2985c89: Log matching property violated. Preceding OpId in replica: term: 9 index: 53. Preceding OpId from leader: term: 10 index: 54. (index mismatch)
I0227 17:27:09.439105 13606 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate c31e774156b64c02922906e3e2985c89 for term 10 because replica is already servicing an update from a current leader or another vote.
I0227 17:27:09.439311 125382 heartbeater.cc:345] Connected to a master server at 127.112.245.252:58453
I0227 17:27:09.439558 53536 consensus_queue.cc:984] T 00000000000000000000000000000000 P c31e774156b64c02922906e3e2985c89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I0227 17:27:09.446537 55954 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:27:09.448315 55954 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
I0227 17:27:09.455088 55957 mvcc.cc:203] Tried to move safe_time back from 6354195369204924416 to 6354195328171925504. Current Snapshot: MvccSnapshot[committed={T|T < 6354195369204924416}]
I0227 17:27:09.471638 55953 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31e774156b64c02922906e3e2985c89. Latest consensus state: current_term: 10 leader_uuid: "c31e774156b64c02922906e3e2985c89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31e774156b64c02922906e3e2985c89" member_type: VOTER last_known_addr { host: "127.112.245.254" port: 35995 } } peers { permanent_uuid: "6cd5de61888340e681a3e5cd19fad418" member_type: VOTER last_known_addr { host: "127.112.245.253" port: 50732 } } peers { permanent_uuid: "2e3d7e621b864672aaa91ea2d0bbd1b9" member_type: VOTER last_known_addr { host: "127.112.245.252" port: 58453 } } }
I0227 17:27:09.471671 55953 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2e3d7e621b864672aaa91ea2d0bbd1b9 [sys.catalog]: This master's current role is: FOLLOWER
W0227 17:27:09.479941 126593 connection.cc:647] server connection from 127.112.245.193:36811 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:36811): Connection reset by peer (error 104)
W0227 17:27:09.479943 124930 connection.cc:647] client connection to 127.112.245.193:55806 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0227 17:27:09.479976 125257 connection.cc:647] client connection to 127.112.245.193:55806 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:55806): Connection reset by peer (error 104)
W0227 17:27:09.480051 124930 consensus_peers.cc:457] T fcd66b3d06b5442b9b1b4d34c6d0667b P 39144add463f4fe89dc1ea42c778d8cf -> Peer 9f6459ebb80643b98e01844aa2fd6338 (127.112.245.193:55806): Couldn't send request to peer 9f6459ebb80643b98e01844aa2fd6338. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W0227 17:27:09.480223 124930 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Broken pipe
W0227 17:27:09.482353 137432 connection.cc:647] server connection from 127.112.245.193:56476 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:56476): Connection reset by peer (error 104)
W0227 17:27:09.482563 124929 connection.cc:647] server connection from 127.112.245.193:35019 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:35019): Connection reset by peer (error 104)
W0227 17:27:09.482897 125258 connection.cc:647] server connection from 127.112.245.193:52057 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.193:52057): Connection reset by peer (error 104)
W0227 17:27:09.483512 13446 connection.cc:647] server connection from 127.112.245.193:47215 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104)
W0227 17:27:09.483717 13446 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Broken pipe
I0227 17:27:09.484450 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 124859
W0227 17:27:09.511212 126594 connection.cc:647] server connection from 127.112.245.194:54775 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:54775): Connection reset by peer (error 104)
W0227 17:27:09.511304 125258 connection.cc:647] server connection from 127.112.245.194:52297 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:52297): Connection reset by peer (error 104)
W0227 17:27:09.511715 125257 connection.cc:647] client connection to 127.112.245.194:55958 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:55958): Connection reset by peer (error 104)
W0227 17:27:09.527729 137434 connection.cc:647] server connection from 127.112.245.194:34489 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:34489): Connection reset by peer (error 104)
W0227 17:27:09.529784 13448 connection.cc:647] server connection from 127.112.245.194:50217 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.194:50217): Connection reset by peer (error 104)
I0227 17:27:09.530779 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-tserver with pid 125074
W0227 17:27:09.551887 126591 connection.cc:647] server connection from 127.112.245.195:35483 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:35483): Connection reset by peer (error 104)
W0227 17:27:09.554136 137434 connection.cc:647] server connection from 127.112.245.195:51016 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:51016): Connection reset by peer (error 104)
W0227 17:27:09.556385 13449 connection.cc:647] server connection from 127.112.245.195:48937 recv error: Network error: failed to read from TLS socket (remote: 127.112.245.195:48937): Connection reset by peer (error 104)
I0227 17:27:09.557909 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 126213
I0227 17:27:09.586350 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 137406
I0227 17:27:09.606333 115671 external_mini_cluster.cc:1131] Killing /data/1/adar/apache-kudu-1.9.0/build/debug/bin/kudu-master with pid 12784
Shutting down hive metastore.
[       OK ] HmsConfigurations/MasterFailoverTest.TestMasterPermanentFailure/1 (109815 ms)
[----------] 4 tests from HmsConfigurations/MasterFailoverTest (220225 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (220225 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] HmsConfigurations/MasterFailoverTest.TestDeleteTableSync/1, where GetParam() = 2

 1 FAILED TEST